Summary: Alan challenges one of the recently published "third party studies" comparing ATI and NVIDIA video technology and shows you just how deep the rabbit hole goes... unless you'd rather go on believing what you want to believe. Will you take the red pill or the blue bill? Alan's claim? The people who don’t care about video quality are the people who don’t understand the differences.
The question is why. Should we even be doing this? No, I don’t mean the question of whether or not we should share links to the community. I’m asking: why are we all trying so hard to get to the truth? Children in grade school can give you the quote “ignorance is bliss” but we can actually understand why that’s true. Imagine if you didn’t know what you do about technology. You would go to Best Buy and get the graphics card with the coolest box, and when you discovered that the 32-bit PCI graphics card you just bought only runs Half Life 2 at 10 fps at 640x480, you wouldn’t get pissed, but would just assume that choppiness was the state of PC gaming. You would be beaming after buying “professional” speakers from two guys in a white van, and you would actually be excited about getting a Dell. Imagine if there was no Internet and no review magazines. You’d be happy just because you didn’t know any better. Like Neo in the original Matrix film, we are all given the choice between taking the red pill or the blue pill -- the choice between learning the truth and facing the consequences of that knowledge or choosing to continue with our lives as we have been, finding comfort in believing whatever we want to believe. I’m going to make you an offer. If you choose to you continue on, I will show you just how deep the rabbit hole goes. I can’t guarantee that you’ll like what you read. Some of you may not even want to believe me or will even try to downplay its significance. All I can offer you is the truth. What you choose to do with it is up to you. If you choose to quit now and stop reading, that’s OK. You’re not going to die. I’m not going to reveal any life-altering truths. You’ll be happy and return to the life you had before you encountered this article. You may even end up spending less money. Lao Tzu once wrote: “To know that you do not know is strength. Not knowing that you do not know is sickness. The sage is sick of being sick, therefore he is well.” As long as you know that this article will be here for you when you’re ready, that’s OK with me. So what your decision? SIDEBAR: Overly elaborate introductions are my specialty.
However, when it comes to video performance and quality, all of a sudden most people suddenly assume that it’s all the same, or that you’d have to be a videophile with a trained eye to see a difference, or that the company that provided the best video quality 10 years ago is still the company with the best video quality today. After all, as long as it’s “ISF certified” it’s good, right? What this article is going to show you is that
So yes, that was an overly elaborate introduction into what will essentially boil down to an article about video quality on the PC, but that was the only way I could have gotten you this far in an article on video quality. The people who don’t care about video quality are the people who don’t understand the differences… You can still choose to quit now and insist that you don’t care. If you give me a chance to show you what I have, I think you’ll find it worthwhile. SIDEBAR: How could the first Matrix movie be so good, and the sequels so silly? Percentage-wise, they did a poorer job than Star Wars.
For each of these questions, make sure you’re looking at the full resolution pictures and not just the thumbnails.
Can you spot the difference? (You can ignore the fact that the images are off by a few frames) Source: Lord of the Rings: The Two Towers. 1080i HDTV broadcast (The WB). SIDEBAR: The next Harry Potter film has a budget bigger than Titanic.
[image]
How about now? Can you spot a difference? Source: Nature: Under Antarctic Ice. 1080i HDTV broadcast (PBS) SIDEBAR: It costs about $8000 per person for a vacation to Antarctica.
[image]
How about now? Can you spot the difference between the processed and unprocessed image? Source: The O.C., 720p broadcast (Fox) SIDEBAR: Most people who buy an HDTV find themselves watching more TV shows just because it's in HD.
[image]
Finally, can you spot a difference between these two? Try the animated comparison tool if you need to. The answers are all on the next page. Source: Lord of the Rings: The Two Towers. 1080i HDTV broadcast (The WB).
When it comes to DVD movies or 1080i HDTV, the ultimate arbiter of image quality ends up being the quality of the video processor which handles deinterlacing and optionally, noise reduction. If you do not deinterlace at all, you may run into the artifacts seen in the first question of the quiz. Believe it or not, I've seen the $12,000 Qualia TV run into problems like this with HD content at the Costa Mesa Sony Style (presumably an issue with improper settings). This is actually a cadence test rather than a pure deinterlacing test, but we can go over that later. The difference between proper deinterlacing and improper deinterlacing can mean the difference between maintaining a full-resolution image and one where half of the data thrown away. If you've heard people say that 1080i looks worse than 720p, it's only because their deinterlacer is throwing away half of the data to produce 540p! (1920x540 vs 1280x720). Movies that are broadcast at 1080i allow full reconstruction of 100% of the data (1080p at 24 fps). This is what is shown by the Antarctica pictures. Although the lower resolution screenshot is still superior to DVD, it's a step back from what is truly available in the source. The third set of test images shows you that even if you're dealing with a very high-quality progressive scan source such as 720p HDTV, a high-quality video processor is still necessary for removing the random noise (grain) that was introduced either in the original source recording or at the time of MPEG-2 compression. The differences between video processors for image quality are big. With TVs getting bigger and cheaper, visual problems are only going to be magnified. Unless you're a person who only watches movies on a Sony PSP, you should care about the quality of the video processor. Deinterlacing 101
When you look at a still image on a PC monitor, plasma or LCD panel, or DLP/LCD projection TV, you are looking at a “progressive scan” image. This means that the display updates all of its pixels simultaneously. The problem is that all DVDs and 1080i HDTV are "interlaced," so they transmit only half of the image, alternating between the odd and even lines. We call each "half image" a field. So for a 720x480 image, the DVD stores it as two blocks of 720x240 pixels.
HQV (Hollywood Quality Video) is a marketing brand-name for Silicon Optix's video deinterlacing technology. Some of you may have remembered me talking about the company in our Ultimate Home Theater PC system build a few months ago. Their technology is based upon Lockheed-Martin's military imaging technology and really represents some exotic stuff. The "HQV Benchmark DVD" was engineered to make testing of video processors easier by having all of the necessary test patterns on a single handy disc. Anytime you have a manufacturer-developed benchmark, it's important to evaluate the test for bias. The HQV Benchmark DVD was designed to showcase the strengths of the Silicon Optix's products. So, if you were able to develop a video processor that was better than Silicon Optix Realta, the DVD would not be able to show you that difference: they'd both get perfect scores even though one may be better than the other in real-life. That's a clear limitation of the benchmark. That said, it turns out that because HQV reflects the top-of-the-line, no-compromise video processor, the tests used to showcase HQV technology end up being an enormous challenge to "normal" video processors such as the kind you find in DVD players that don't cost $3000, or video cards from ATI and NVIDIA. So for these types of products, the HQV Benchmark DVD is a great and honest test that brings objectivity to video quality. That's why the benchmark has been adopted by a wide range of testers, ranging from the techies at AVSForum.com and Audioholics.com to the reviewers at Widescreen Review and the Perfect Vision. I know that NVIDIA's PureVideo team has used the benchmark in-house, and I'm sure ATI's team has as well. Recently NVIDIA commissioned an independent 3rd party to use the HQV Benchmark DVD to compare NVIDIA's performance against ATI's. (Click for PDF document here). There's nothing wrong with a company commissioning a study. If NVIDIA believes in the superiority of their technology over the competition, why not try to get the word out? The problem is that the HQV Benchmark DVD is only good when the reviewer is scoring the tests properly. Although the company has a distinguished record when it comes to evaluating traditional PC hardware such as CPUs, Principled Technologies has made some significant errors in this commissioned study. I'm not sure how much Principled Technologies understands video technology since they inexplicably talk about "Dolby Noise Reduction" when it comes to video (Dolby produces no such technology; they meant Digital Noise Reduction). They also tested GPU video processing quality by outputting 1080i DVI into a Samsung LCD-TV with a native resolution of 1280x720, which means that their DVI results are confounded by the video processor in the Samsung TV. Worse, Principled Technologies occasionally came up with what can be called creative interpretation of how a few image quality tests are scored. So for the second half of this article, I'm going to go over the HQV Benchmark DVD and explain how it should be used. Instead of just focusing on the hard details of the benchmark, I'll also be going over the basis for each test which will help you understand video processing technology and why the test is needed. When appropriate, I will point out where the NVIDIA commissioned document from Principled Technologies has erred. Why do I understand the HQV Benchmark better than Principled Technologies? Well, for one, I've been hired by Silicon Optix to help with version 2.0 of their benchmark – so I know the strengths and weaknesses of the current version. I have also been providing feedback to the company based upon their prototypes, so I have a HQV video processor myself. With my digital photography and imaging background, this means that I know what technology Silicon Optix uses to achieve a perfect score as well as what 3rd-party software tools can achieve the same look through different algorithms. In contrast, Principled Technologies arbitrarily came up with their own criteria based upon "discussion amongst testers" rather than obtaining the readily-available Denon DVD-5910. You can take my article with a grain of salt given my own involvement with Silicon Optix. That said, I will let the screenshots speak for themselves and you can run your own tests with the HQV Benchmark DVD. Likewise, you could say that FiringSquad is technically a competitor to Principle Technologies and that reflects an ulterior motive. That's not an ulterior motive: I'd just like to keep companies honest whenever I have time to write. So let's dissect the HQV Benchmark DVD. SIDEBAR: Windows XP Media Center Edition 2005 systems allow you to use two HDTV tuners to record two shows concurrently.
[image]
Color Bar / Vertical Detail
The first test on the DVD is the "Color Bar / Vertical Detail" test. It turns out that this does not actually evaluate detail. That test comes later. Instead this tests how well a video processor can distinguish between moving images and still images. Things in motion should be deinterlaced in a different manner from things that are not moving.
[image]
Counterpoint
Well, what happens if the director was recording flicker and was filming an object that alternated between white and black at 60 Hz? What would happen? It turns out that the data on the DVD would be identical. Playing devil's advocate, you could argue that HQV and everyone else with a sharp image are doing the wrong thing. Flicker is good. This is an ambiguous test in that both sides would have technically accurate statements. In practice, this is a bogus claim made only by companies that fail the test. Appropriate Scoring
This test evaluates whether a de-interlacer can properly determine whether things are in motion or things are static. This is important because video processors need to apply different algorithms depending on the source.
The next three tests of the HQV Benchmark DVD evaluate how well the video processor does with moving objects. The first "jaggies pattern" is a simple spinning bar, the second "jaggies pattern" is a set of three bars with a waving motion (like waving your hand), and the third test is a real-world picture of the American flag in the wind. These all represent scenes that any respectable deinterlacer should recognize as being in motion. If a deinterlacer passed the first test only because it always assumed that objects were static, these test patterns would reveal feathering artifacts and result in a failing score at any degree of rotation. The only method for dealing with motion is to throw away some of the pixels that would cause feathering (at least when it comes to products costing less than $60,000). So, the difference between good and bad video processors is how selective it is at throwing away data. If you only throw away the pixels that would cause feathering, you maximize as much detail as possible. Remember, this is the "treatment" phase of the problem – it's only as good as the accuracy of detecting which pixels have the artifacts. When you throw away data, you can replace it by averaging the pixels above and below the image. This loss of resolution causes jagged edges to form, most prominently on diagonal lines. Higher quality deinterlacers try to address this. This can be done with diagonal filtering or some sort of more intelligent reconstruction method that's more involved than just averaging the pixels above and below. Minimizing the amount jaggies gets increasingly difficult as the angle becomes more oblique. Scoring of the jaggies tests takes this into account. [image]
[image]
[image]
[image]
So all three of these tests work together to ask
Some video processors throw away half of the entire image data even if only one small part of the screen is in motion. To check for this, reviewers need to look for jaggies at the logo. Any processor that causes jaggies at the logo is throwing away more data than it should and only gets a score of zero. The equivalent check in the flag clip is the brick wall where you can check for loss of detail. Counterpoint
These tests are conceptually unambiguous in that there is no reasonable argument to suggest that these are flawed. Of note, the speed of the rotation and the amplitude of the waving in the HQV Benchmark DVD are designed to maximize the amount jaggies. This makes it easier to identify what video processors are doing. There are some manufacturers who fail this test, but have developed their own video clip that looks like the HQV Benchmark, but in fact has been carefully crafted so that it's easier to pass the test by reducing the speed of the bar or changing the size of the yellow zone. Beware of any similar looking tests that do not have the HQV logo. Appropriate Scoring
This test evaluates what a video processor does once it determines that things are in motion. Any time things get blurry, or you get jaggies, the video processor is throwing away data. Sharp lines mean that the video processor incorporates diagonal filtering (Faroudja, Gennum, HQV) or some sort of advanced jaggy-prevention technology (NVIDIA PureVideo). It's worth noting that although NVIDIA scores 3 points in this test, they fail at the very last frame of the yellow zone. An optimist would say that they're only one frame away from getting 5 points.
Waving Flag: 0 – Jagged lines are present in the stripes of the flag 5 – Jagged lines are occasionally present (i.e. when the stripes of the flag are most oblique) or the background is soft 10 – The red and white stripes are smooth and fine detail is seen. [image]
In this test, a video clip of a cityscape is used to evaluate for picture quality. The video provides several areas to look for detail such as in stairs, a grass field, or a bridge. Basic video processors do not incorporate any sharpening tools at all. These deserve a score of zero. This is what is meant by a "soft image." Most video processors including those from NVIDIA and ATI fall into this category. In this regard, NVIDIA and ATI should only receive 0 points on this test rather than the 5 that were awarded by Principled Technologies. Importantly, a score of zero does not mean that the product is blurring the image. It means that the playback device is outputting exactly what is on the DVD. [image]
The simplest sharpening algorithms will restore some of the detail, but it may generate artifacts such as sharpening halos. The appropriate score for video processors with sharpening tools that cause artifacts is 5 points. NVIDIA's monitor-wide sharpening slider (under Color Correction tab in the NVIDIA settings) falls under this category – it would only be reasonable to award NVIDIA the 5 points if the level of sharpening could be selectively applied to the overlay video or VMR only. [image]
The most advanced sharpening algorithms can restore detail without causing artifacts. There are several methods to achieve this effect. Here is an example of a software-based approach: [image]
Counterpoint
HQV has decided that bad sharpening (5 points) is better than no sharpening (0 points). There are arguments for and against this distinction. I don't agree with this entirely. Sharpening can be performed at any stage of the video processing chain. When testing a "system" with HQV (i.e. monitor + DVD player), this scoring system makes a lot of sense. You need some sharpening. That said, sharpening on the TV is usually better than sharpening on the DVD player because it can be optimized for the size of the screen (Note that I'm talking about $200 DVD players – once you get to the DVD players that are $1k or more, things change). Appropriate Scoring
For the purposes of consistency across all reviewers, the appropriate way to score is
To understand the basis of these tests, you need to understand how noise reduction works. You can either filter out noise spatially (i.e. blur the image slightly) or filter it out temporally by recognizing the fact that noise is random, and removing noise by averaging out the pixels that have changed from frame to frame. Temporal filters are better because they theoretically preserve more detail than simple spatial filters. The first "Noise Reduction" test evaluates noise by using video containing slow camera pans. These images are recorded using analog broadcast tape (i.e. what TV studios may have used previously) which provide a large amount of baseline noise. [image]
Products with no noise reduction technology should receive zero points. This reflects current PC products. Products with traditional spatial noise reduction will usually get 5 points because significant details will be lost. The only way a noise reduction algorithm would receive zero points is if the noise filter was some sort of global blur. [image]
Video processors that reduce noise without loss of image detail are awarded 10 points. This includes temporal noise filters or advanced wavelet-based or similar type spatial noise filters, or a combination of spatial and temporal filtering. Wavelet-based spatial noise filters are not found in any shipping consumer hardware product. We have used such a noise filter in software for this simulation: [image]
In this test, you have video of a roller coaster. A temporal filter that is blindly applied will not be able to distinguish the movement of the roller coaster from random noise – it will see both as "changing pixels." This results in a ghosting or "echo" of the moving roller coaster because it's trying to blend out some of those pixels. A zero score is given when noise is present (a product which has no filtering) or when temporal filtering is blindly applied resulting in motion trails behind the rollercoaster. None of the video cards tested by Principled Technologies provide any noise reduction and they should all have received zero points in this category. It is unclear what Principled Technologies thought they were seeing when comparing the Intel 945G against NVIDIA products. A score of 5 is given to video processors that reduce noise at the expense of significant detail. This includes traditional spatial noise filters. In the case of moving objects, a dumb spatial noise filter does a better job than a dumb temporal noise filter. This scoring system allows a video processor that lacks the horsepower for motion-adaptive temporal noise reduction to still score well on the HQV Benchmark DVD as long as it switches between spatial and non-adaptive temporal filters based upon the content. A score of 10 is given to noise filters which are able preserve detail even when the scene contains moving objects. This includes motion-adaptive temporal filters. Motion-adaptive temporal noise filters distinguish between random noise and moving pixels. When this roller coaster clip is played back through an HQV processor, you get the benefits of temporal filtering (minimum noise and maximum detail in the sky and track) without ghosting or motion trails behind the roller coaster. Again, some sort of advanced wavelet-based spatial filter would also be able to earn a score of 10. Counterpoint
Principled Technologies claims that these noise reduction tests "have little bearing on evaluating the abilities of a graphics adapter or DVD player" and that noise only reflects copies of video tapes or analog broadcast interference. They are wrong. Many remastered DVDs incorporate some sort of noise reduction but they still have noise that is introduced in the MPEG-2 compression stage. I think noise reduction is one of the most critical elements of a high-end video processor. It makes a substantial difference even with 720p HD broadcasts. Noise reduction is particularly important for big screen TVs as well.
[image]
[image]
You can draw your own conclusions on whether or not modern DVD and HDTV content have noise. I want you to trust your eyes on seeing where noise reduction helps and where it doesn't. Appropriate Scoring
Noise Reduction Motion-adaptive noise reduction
0 – No noise reduction is applied, or the noise reduction produces artifacts
Remember when I talked about DVDs and 1080i storing interlaced video and how each field was recorded 1/60 seconds apart? There's are exceptions to that, big ones at that. For Hollywood movies recorded at 24 fps on silver halide film, or 30 fps video recorded on high-end progressive scan HD-DVCAM camcorders, or animation, the full detail can be recorded. The difference is that these sources are recorded as progressive images and then interlaced for distribution on DVD or 1080i HDTV rather than being interlaced recordings. You should always be able to encode any source into 60 fields per second while preserving all of the detail as long as the source was a) recorded as progressive and b) had a frame rate below 30, right? Yes. That's exactly right. Let's look at 24 frames per second film. If you divided them up, you could create 48 fields in a second where each pair of fields is recorded at the exact same time! Now, if you needed to take those 48 fields and spread them across 60 fields, you could do it with a 2:3 ratio. If your frames were A,B,C,D then you could both halves of A and both halves of B in the first 2 frames (4 fields). In your third pair of fields, you could have half of B, and half of C, and in the fourth pair of fields, the remaining half of C, and first half of D. In the fourth pair of fields, you could repeat both halves of D. This represents a 2:3 cadence because 2 fields of one frame are shown and then 3 fields of the next. The ability to reconstruct images from video encoded in this format is called 3:2 pulldown. Believe it or not, the ability to detect this cadence is not trivial especially if a scene starts somewhere in the middle instead of the "A field." The ability to deal with tougher sequences is sometimes called bad-edit detection. Any time you fail to recognize the cadence, you end up throwing away up to half of the resolution. Suffice it to say, cadence detection is important for maximum detail. [image]
In this test, a video of a racecar is shown. This is taken from a real film in which the cadence does not begin at the "A" frame. Video processors that fail to recognize this cadence end up throwing away half of the resolution, which in this case generates the moiré pattern on the bleachers. [image]
This test also checks how fast a video processor is at detecting the cadence. Even though you started at a spot other than A, you should get back into sync within just a few frames. It turns out that a lot of processors have trouble recognizing this pattern effectively and may take more time before it realizes the appropriate cadence. In this case, the position of the race car helps you determine how long it takes to detect the signal. 10 points is given if the video processor can lock-on before the race car is one car length before the HOMESTEAD billboard (5 frames). 5 points is earned if the processor needs more time, but can switch within half a second. This means that the video must lock on as the front right tire (from the perspective of the driver) of the racecar reaches the first arrow to the right (as viewed by the reviewer) of the HOMESTEAD sign. 0 points is given if the processor has not locked by the time the race car reaches the HOMESTEAD sign. No distinction is made between products that never lock-on versus those that are late. SIDEBAR: The Fantasie-Impromptu has a 4:3 cadence between the right and left hand.
While 3:2 is the most popular cadence used, it's not the only cadence. Documentaries shot on high-speed film run at 30 fps resulting in 2:2. DVCAM or HDV camcorders may use 2:2:2:4 or 2:3:3:2. Movies broadcast on TV may drop every 13th field in order to leave more time for commercials resulting in 3:2:3:2:2. Animation ranges anywhere from 8 to 12 fps. The next sequence from HQV tests all of the 8 major cadence sequences, awarding 5 points for correctly passing the test. Any time you fail to detect the appropriate cadence, you end up losing half of the resolution so scoring is based upon whether or not the video processor throws away of the resolution or whether the full image is displayed. [image]
[image]
[image]
Counterpoint
HQV was engineered as a no-compromise video processing solution, and one element is their cadence detection algorithm which detects "anything and everything." It's no surprise that they would assign 30% of the total score to this feature. Appropriate Scoring
5 – Flickering, jaggies, or loss of resolution absent
In order to pass this these successfully, a video processor must selectively apply different deinterlacing algorithms depending on the region of interest. This tests the "adaptive" nature of the de-interlacer. 10 points is given if the text is sharp and the background is clean. 5 points is given if the video deinterlacer defaults to "video" meaning that text is clear but the rest of the screen loses detail 0 points is given if the video deinterlacer remains in "film" meaning that the movie is clear, but the scrolling text produces feathering or combing artifacts. [image]
[image]
These images were provided by Silicon Optix Counterpoint
In that regard, it's somewhat ironic to read through Principled Technologies' commissioned study. They've made several errors in evaluating the video playback performance of PC GPUs resulting in 10 points of artificial inflated scores for NVIDIA and an artificial 5 point penalty for the i945G. In my testing, NVIDIA PureVideo solutions playing a DVD via a native 480p window using a 1280x1024 DVI connection resulted in only 51 out of 130 points. This is identical to the results obtained at Anandtech.com but clearly different from the 86 points that Hot Hardware gave or the 68 point that NVIDIA’s commissioned article gave. The irony is that even though NVIDIA's scores were inflated by Principled Technologies, their "key findings" are ones I still agree with completely. Namely, that NVIDIA PureVideo is superior to ATI's Catalyst 5.8, that PureVideo is superior to basic DVD players, and that PureVideo minimizes some of the common artifacts seen with de-interlacing. In comparison to higher-end DVD players such as the Denon DVD-2910 ($700 retail), NVIDIA still has a ways to go. Likewise, PC manufacturers in general are behind the consumer electronic manufacturers when it comes to capabilities such noise reduction or detail enhancement. I hope that companies like ATI and NVIDIA who are serious about image quality will consider adding high quality noise reduction and detail enhancement technology in their next generation products. The state of PC video quality is also going to change. ATI's Catalyst 5.9 drivers feature a brand-new de-interlacing technology which may address some of the limitations of older ATI products. ATI's AVIVO supports noise reduction with 3rd party TV tuners but it's not yet clear if ATI will be able to implement noise reduction for DVD sources, or better yet HDTV sources. I have not yet had the opportunity to test Catalyst 5.9 myself, so I can't make any comment on how it compares to PureVideo but it warrants a closer look. We at FiringSquad hope to take on that challenge. As you can see, this is just part 1 of our series. Our next step will not only be to test the best from ATI and NVIDIA against the best standalone consumer DVD players when it comes to DVD content, but also to challenge the GeForce 7800GT and Radeon X1800 in tests of high-definition deinterlacing, where the processing demands can be as much as 9 times higher. A Special Treat
The HQV Benchmark DVD has been distributed to various journalists, and consumers can buy the disc for about $30 ($20 with a coupon). While it'd be better if the benchmark were free or advertiser-supported, someone has to pay for the licensing costs associated with the video clips in the Benchmark DVD. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| © Copyright 2003 FS Media, Inc. |