Benchmarking for games

Gaming

To be vendor-neutral you'd have to look at the all the major hardware first, figure out what's commonly supported, and then start programming. Almost no game developer takes this approach in the strictest sense, because the completed game wouldn't harness the steady improvement in hardware over the year or two the game was in development. What really ends up happening is that the artists and programmers let their creative ideas flow, typically overshooting at first and then editing what they have already done to make it work. It's the difference between thinking in English versus thinking in a foreign language and then translating it into English.

Since a useful synthetic benchmark should mimic real games, the best synthetic benchmarks shouldn't be vendor-neutral. Think of Half-Life 2 and Doom 3; a vendor-neutral synthetic benchmark would not be able to capture those differences. What you really want is a synthetic benchmark that has multiple personalities. You want a synthetic benchmark that captures the essence of the games of today and tomorrow.

That's what 3DMark05 does. 3DMark05 is an effective tool because it is very much vendor-specific, allowing the user to utilize intermediate-precision shaders, NVIDIA specific shading models, or even selecting between version 2.0 and 3.0 DirectX Shader Models. In the default mode, 3DMark05 does the best it can to provide one number that reflects overall performance of a typical game of the future. Through the permutation of options, a talented reviewer can use that same tool to simulate even more types of real game environments. That's cool.

What about the 3Dc issue?

In the perfect world, 3DMark05 would add 3Dc support giving it the additional personality it needs to emulate the set of games that implement ATI technologies. So why didn't Futuremark include 3Dc if it would make 3DMark05 a better product?

It comes back to the Futuremark's Benchmark Development Program (BDP). The BDP includes all of the major graphics companies such as ATI, NVIDIA, PowerVR, S3, and XGI. In order to produce a benchmark that remains useful for two years, Futuremark needs input from hardware manufacturers and knowledge of their future product plans. To determine what graphics techniques will be adopted, they also get feedback from game developers and review the academic literature. Once they've got all the data, they decide their course of action. Simply put, Futuremark determined was that 3Dc was not supported by enough developers and/or future hardware that warranted investing their development costs of adding 3Dc or the extra ~150MB into the download. Moreover, 3DMark05 already features the DirectX version of normal map compression: DXT5 (also the approach used by Doom3).