Problem 1: Innovation Bias

For many gamers, the job of professional video game reviewer seems like a dream. You play games all the time and you get paid good money to write about it. It's like being a radio DJ or winning the lottery. You don't have to go through any special education like a lawyer or doctor and you pretty much can work on your own schedule. Playing through a hundred games a year causes problem. It's easier to get jaded when all you do is play games in and out, when it's pretty reasonable to assume that at least half of those games are boring or frustrating.

As a result of being jaded, reviewers end up falling into the trap of innovation bias. Bruce Shelly of Age of Empires III talked about innovation bias as being one of the problems of the industry. He's only partially right. AOE3 is necessarily a polarizing game because resource micromanagement is such a large portion of the game. You'll either find it interesting or boring, and that accounts for the wide range of scores. For a lot of people, it's just not that fun. The problem is that if you read through most of the reviews of the game, you'll see that the negative aspects aren't about how the game requires too much micromanagement, but how the game feels just like AOE2 and doesn't bring anything new to the table. The problem is that for gamers who never played AOE2, AOE3 would actually be a great game to get. I can count on one hand the number of reviewers who recognized that. But AOE3 isn't a good example. In this case innovation bias means that some good games get lost in the crowd – a stellar game is still going to be noticed, and as long as there are about a dozen of those a year, there'll be plenty of choice for the 76% of you who buy up to 6 games a year.

The problem arises from a game like Project Gotham Racing 3. It is good? Sure. I don't think it's as fun as a game like Need for Speed: Most Wanted, but it's still a game that earned 90% at FiringSquad. On the other hand, if you look at the way the game is reviewed by most video game sites, a huge deal is made about the in-cockpit views because it's innovative. The cockpit view for "street cars" isn't something new – Square did it with the PS2 launch game Driving Emotion S. PC racers such as Grand Prix Legends have also done in before. Very few reviewers actually talk about the flaws of the cockpit view in PGR. The most egregious being that it’s difficult to read the dials because the steering wheel is in the way or the lighting is a problem. Too much gets made about the cockpit view in PGR3 and not enough attention is paid to the problems of the game.

Even for my own reviews – are the police chases in Need for Speed: Most Wanted that good, or am I blinded by the novelty and innovation of the feature? Am I criticizing the length of Ridge Racer 6 because I feel like the game has already been done and brings little to the table other than nice graphics?

There's not much that you can do other than try to be aware of a reviewer's innovation bias. At the end of the day, it's still important that reviewers play a wide range of games. It's just something you have watch out for. If a review spends too much time talking about how it's no different from the prequel or spends too much time talking about a single feature, take pause.

There was another area where “being jaded” comes into play. How come I didn’t have any screenshots of real-tracks from PGR3? It was all about high-resolution shots from the Garage mode… The vast majority of game reviewers stick with direct framebuffer grabs which may not reflect the actual experience and usually, we just take a hundred screenshots and hand-pick the more stylish ones we can find. The only time I’ve ever used direct framebuffer grabs is with GT4 and even then I spent a fair amount of time with a disclaimer.

For the record, Jakub completely disagrees with me about Innovation Bias. He doesn't see it as a problem but rather a reflection of the human desire for constant improvement. I think that you can have a desire for constant improvement but still be affected by innovation bias.