Athlon 64 Vs. Pentium 4 article: On the Justification for..

[I sent this letter to Hardocp, Tomshardware, and Anandtech. Comments
welcome -- rms]

Hi, this is about your Athlon 64 Vs. Pentium 4 article, specifically the use
of Quake3 as a CPU benchmark when comparing AMD vs. Intel cpus, as shown on
this page

http://www.hardocp.com/article.html?art=NTI0LDU=

http://www.anandtech.com/cpu/showdoc.html?i=1884&p=13

http://www.tomshardware.com/cpu/20030923/athlon_64-22.html#opengl_benchmarks

Let me say the article is great, no complaints there. I know it takes alot
of work to produce these articles.

Now, I see two reasons for using a game as a cpu benchmark:

1) It presents a fair (emphasis on the word 'fair') comparison of the
competing cpu architectures and scaling issues.

2) The game itself is of current interest to the community.

In your article you already concede 2). Quake3 itself is not relevant as a
game to anybody, and framerates in the 400's make the results meaningless to
gamers. Quake3-derived games are another matter, and are still popular and
certainly relevant. More on these later.

I believe there is strong evidence that Quake3 does not provide a fair
benchmark for comparing *modern* (AthlonXP and possibly Athlon64 as well)
AMD cpus vs Intel cpus. The reason being (and let me emphasize that I don't
know this as an verified fact, I'm going on what a couple of programmers
involved with helping AMD produce optimized game code have told me) that the
Quake3 cpu recognition code does not recognize the AthlonXP as an
SSE-capable cpu. Not only that, but the 3DNow code in Quake3 is apparently
non-functional for this cpu.

The politics and history behind this are interesting, but probably boil down
to the AthlonXP being released well after Quake3, and Carmack being rightly
uninterested in patching an old game.

If this is true, you are benchmarking two equally SSE-capable cpus against
each other, using a game engine which enables SSE for the Intel cpu and
*disables* SSE for the AMD cpu (apparently there's no simple way to force
SSE recognition either that works), for no valid reason, other than the game
is too old to know about the AMD cpu's capabilities. What would be even
worse is if this same recognition problem carries over to the Athlon64 (I
have no word on this) and to newer Quake3-based games.

Again, assuming this is true, it removes any rationale for using a 3-year
old game that: a) few people play, b) which gives ridiculously high scores,
and which c) unfairly handicaps AMD cpus; as a benchmark to be used
specifically in comparing AMD cpus vs their Intel competitors in articles
such as this one.

So. Here are the recommendations I, as an interested Hardocp/Anandtech/Toms
reader (and admitted AMD fan) am making to you and your site:

1) Investigate this matter further, and write an article discussing it. And
in particular discuss the relevance of this cpu issue to current
Quake3-based games. Assuming there is in fact an Intel bias to Quake3-based
benchmarking I think people would be very interested to learn about it.
Apparently the SSE issue does indeed carry over to later games.

1) Assuming there is a bias, discontinue using Quake3 as a cpu benchmark,
and especially discontinue it's use when comparing AMD vs Intel cpus. The
game will never be patched to fix this issue, and using 3rd party fixes
noone cares about is more or less pointless too. I'm referring to the dlls
on this page:

http://speedycpu.dyndns.org/opt/

This guy is one of the programmers I referred to earlier, and he tells me
the dlls do not enable SSE where it really matters anyway. The other was a
student working at AMD writing assembly 3DNow code. The best solution is
simply to retire this benchmark, just as Q1 and Q2 were retired.

rms

rms wrote:
 > [I sent this letter to Hardocp, Tomshardware, and Anandtech. Comments
 > welcome -- rms]
 >
 > Hi, this is about your Athlon 64 Vs. Pentium 4 article, specifically the
 > use of Quake3 as a CPU benchmark when comparing AMD vs. Intel cpus, as
 > shown on this page
 >
<font color=purple> > <a style='text-decoration: underline;' href="http://www.hardocp.com/article.html?art=NTI0LDU=</font" target="_blank">http://www.hardocp.com/article.html?art=NTI0LDU=</font</a>>
 >
<font color=purple> > <a style='text-decoration: underline;' href="http://www.anandtech.com/cpu/showdoc.html?i=1884&p=13</font" target="_blank">http://www.anandtech.com/cpu/showdoc.html?i=1884&p=13</font</a>>
 >
 >
<a style='text-decoration: underline;' href="http://www.tomshardware.com/cpu/20030923/athlon_64-22.html#opengl_benchmarks" target="_blank">http://www.tomshardware.com/cpu/20030923/athlon_64-22.html#opengl_benchmarks</a>
 >
 > Let me say the article is great, no complaints there. I know it takes alot
 > of work to produce these articles.
 >
 > Now, I see two reasons for using a game as a cpu benchmark:
 >
 > 1) It presents a fair (emphasis on the word 'fair') comparison of the
 > competing cpu architectures and scaling issues.

When playing Quake3.

 > 2) The game itself is of current interest to the community.

Not really.

 > In your article you already concede 2). Quake3 itself is not relevant as a
 > game to anybody, and framerates in the 400's make the results meaningless
 > to gamers. Quake3-derived games are another matter, and are still popular
 > and certainly relevant. More on these later.

Indeed, but that depends on what newer features are added. If it's merely
polygon count and more lights in the sty;le of the original then yeah, but I
suspect far more is going on that in some games. As soon as you start using
features (especially newer GPU intensive ones) the old game engine no longer
reflects the current engine.

 > I believe there is strong evidence that Quake3 does not provide a fair
 > benchmark for comparing *modern* (AthlonXP and possibly Athlon64 as well)
 > AMD cpus vs Intel cpus. The reason being (and let me emphasize that I
 > don't know this as an verified fact, I'm going on what a couple of
 > programmers involved with helping AMD produce optimized game code have
 > told me) that the Quake3 cpu recognition code does not recognize the
 > AthlonXP as an SSE-capable cpu. Not only that, but the 3DNow code in
 > Quake3 is apparently non-functional for this cpu.

Essentially, yeah. I'm assuming the Athlon XP is being recognised as an
Athlon, which did not have SSE.

 > The politics and history behind this are interesting, but probably boil
 > down to the AthlonXP being released well after Quake3, and Carmack being
 > rightly uninterested in patching an old game.

Indeed.

 > If this is true, you are benchmarking two equally SSE-capable cpus against
 > each other, using a game engine which enables SSE for the Intel cpu and
 > *disables* SSE for the AMD cpu (apparently there's no simple way to force
 > SSE recognition either that works), for no valid reason, other than the
 > game is too old to know about the AMD cpu's capabilities.

This is only relevent if you are using Quake3 to benchmark the architectures
and attempting to use that comparison as a way of predicting performance
elsewhere.

 > What would be
 > even worse is if this same recognition problem carries over to the
 > Athlon64 (I have no word on this) and to newer Quake3-based games.

Indeed. If however it does carry over (They do not use SSE code on
AthlonXPs) then Quake3 may be considered as a suitable benchmark.

 > Again, assuming this is true, it removes any rationale for using a 3-year
 > old game that: a) few people play, b) which gives ridiculously high
 > scores, and which c) unfairly handicaps AMD cpus; as a benchmark to be
 > used specifically in comparing AMD cpus vs their Intel competitors in
 > articles such as this one.

Exactly. Using a 3 year old game to indicate the performance on any newer
features is useless, pointless and unnecessary.

 > So. Here are the recommendations I, as an interested
 > Hardocp/Anandtech/Toms reader (and admitted AMD fan) am making to you and
 > your site:
 >
 > 1) Investigate this matter further, and write an article discussing it.
 > And in particular discuss the relevance of this cpu issue to current
 > Quake3-based games. Assuming there is in fact an Intel bias to
 > Quake3-based benchmarking I think people would be very interested to
 > learn about it. Apparently the SSE issue does indeed carry over to later
 > games.

Good plan.

 > 1) Assuming there is a bias, discontinue using Quake3 as a cpu benchmark,
 > and especially discontinue it's use when comparing AMD vs Intel cpus. The
 > game will never be patched to fix this issue, and using 3rd party fixes
 > noone cares about is more or less pointless too. I'm referring to the dlls
 > on this page:
 >
<font color=purple> > <a style='text-decoration: underline;' href="http://speedycpu.dyndns.org/opt/</font" target="_blank">http://speedycpu.dyndns.org/opt/</font</a>>
 >
 > This guy is one of the programmers I referred to earlier, and he tells me
 > the dlls do not enable SSE where it really matters anyway. The other was a
 > student working at AMD writing assembly 3DNow code. The best solution is
 > simply to retire this benchmark, just as Q1 and Q2 were retired.


Since the game uses few features that newer games use, you cannot use it as
an indicator of performance in the newer games. I don;t think that this is
the reason they use it... I suspect the reason they include Q3 benchamrks is
because everyboady had Q3, many still have it, many know what their old
cards could do and are interested to find out what the new cards can do.
400FPS is insane and shows you just how far cards have come since the game
release, in terms of raw feature speed.

It's much harder to compare the performance in terms of image quality than
it is in terms of FPS. Including an old game that excludes much of the
eye-candy allows easy direct comparison of cards over 2-3 generations. You
also can't really make any comparisons of video cards from scores 2 years
ago as the increase on CPU speed, memory bandwidth etc are likely to make a
difference.

It all comes down to what you think the benchmark is telling you. If you
take the facts (Quake3 on hardware x) fine. If you try to say much else,
you need to be very careful and in most cases you simply cannot say very
much else than hardware x plays it at this speed.

Ben
--
I'm not just a number. To many, I'm known as a String...<!-- ~MESSAGE_AFTER~ -->