Real-world testing (cont’d)

Why “real-world” now?

This is the question I’ve been wondering, particularly in light of the launch of upcoming next-gen GPUs as mentioned on the previous page. In all honesty this probably goes beyond the premise of this article, which was to explain our methodology and why we test hardware the way we do, then I saw this post by Kyle in his forums where he suggests that this is somehow a new phenomenon that he just now realized:


Using "gaming benchmarks" for CPU testing is broken in its current state. What worked a decade ago, does not work now to show anything applicable to gaming performance as it once did. It does not work any more to reflect any sort of real world benefits.


The fact of the matter is this is simply not true. Gaming performance when testing at high-res has been bound by the GPU rather than the CPU dating all the way back to when the first GPUs began to offload more work from the CPU. I can say this with complete 100% certainty because we’ve been conducting both high resolution and low resolution performance results in our CPU reviews for some time.

You can go back to our CPU reviews from a year ago, or seven years ago and find a mixture of both high and low-res testing. In those reviews you’ll find at high res the margin separating the various CPUs always decreases by the time we’re running at high-res.

Are "real-world" CPU benchmarks useless? No, they're a perfectly good illustration for new readers that video cards can be a bottleneck for CPUs at high resolutions. But the moment a better video card comes out, or someone is interested in possibly CrossFire/SLI results, these "real-world" benchmarks become irrelevent. Let's be honest, when we want to know how many pixels a video card can push, we read video card reviews.

[H]ardOCP’s numbers are very good at presenting Core 2’s performance for a very specific audience: those users with a Core 2 Extreme X6800, Core 2 Duo E6700, or AMD Athlon 64 FX-62 who happen to have a GeForce 7900 GTX and also game at high screen resolutions and high image quality settings, other than that, they don't tell you anything else. What if you run lower resolutions? What if you're going to buy a G80 or R600? What if you have SLI or CrossFire?

Instead of trying to replicate “real-world” usage we tried to provide a mixture of both tests that aren’t GPU-bound (the 800x600 results in our Core 2 review) to better illustrate the performance between the different CPUs, as well as high-resolution 1600x1200 results with 4xAA/8xAF in order to demonstrate usage more typical of a high-end gamer. We also included CrossFire results at 1600x1200 and 2048x1536 to give dual-GPU users an idea of how CPU performance scales from single-GPU config to a dual-GPU config.

Calling this method of testing “virtually worthless” or suggesting that we’re somehow “lying” to our readers is not only misleading to the public, it’s downright unfair to us and other websites.