3DMark (cont’d)

Indeed, the industry-standard DXT5 can provide similar bandwidth savings to 3Dc. Depending on whether or not you normalize the data, DXT5 can even provide a faster solution than 3Dc. True, 3Dc offers normal maps with less compression artifacts than DXT5, but if a developer wanted to implement "anti-aliased lighting" via mipmapped un-normalized normal maps, he couldn't do it with 3Dc. So it's a feature with trade-offs. Of course, should things radically change in the world of 3Dc, there's nothing to stop a "3DMark05: Second Edition" from being produced.

Still, the fact that something like 3Dc won't be universal doesn't change the fact that a handful of superb games are going to feature the technology. Without 3Dc, 3DMark05 alone will not be a good marker for games like Half-Life 2, Serious Sam 2, or Tribes Vengeance running on ATI cards. But that's OK. A good reviewer will know this and take this into account. 3DMark05 isn’t the one-stop solution for benchmarking, but it is an additional tool in the armamentarium of the reviewer, and a well designed one at that.

That is the core of our job as reviewers though -- we'll try to sort out the benchmarks we need to run and the right way to interpret them. Our goal is to help you decide what the best product is for you… and what a coincidence, that's the same purpose of a benchmark. You don't want a vendor-neutral benchmark. What you really want is a vendor-neutral reviewer with realistic benchmarks!

Open Source Benchmarks?

The concept of realistic benchmarks limits the value of open source benchmarking. It simply is not the end-all ideal solution. If you take on the task of creating an open source benchmark mimicking the process of camera RAW image using something such as dcraw, it is unclear that it would truly be any better than a SiSoft Sandra type of synthetic test. This is because the raw processors in active use by real photographers will have vastly different performance. Knowing that a specific product is faster on an open source RAW developer is of no importance to a photographer who uses Capture One on a day to day basis.

Fixed Benchmark Suites?

Fixed benchmark suites such as Bapco’s SyMark are common in the mainstream press. The appeal seems obvious – SysMark attempts to mimic “real-world” tests in the same way I’ve described above. There’s just one catch though, these fixed benchmark suites do not age well. If you look at the applications used, they’ll be several generations old. SysMark 2004 uses Photoshop 7.01 for example. Performance in Photoshop 7.01 isn’t necessarily predictive of the performance you get with the current Photoshop CS2.

So, when you look to website or magazine for hardware reviews, start your reading with a place where you trust the software reviews too. Anybody can get a bunch of hardware together and show you pretty graphs. That doesn't mean that they understand how those numbers translate into real performance for games you actually play or the applications you actually run. You need someone passionate about the same software you are passionate about.

Alright, let’s look at the test suite I came up with for the 2005 Eternal Battle.