Forgot your password?

Review Scores the "Least Important Factor" When Buying Games 169

Posted by Soulskill
from the arbitrary-numbers-are-arbitrary dept.
A recent report from a games industry analyst suggests that among a number of factors leading to the purchase of a video game — such as price, graphics and word of mouth — the game's aggregated review score is the least important measure. Analyst Doug Creutz said, "We believe that while Metacritic scores may be correlated to game quality and word of mouth, and thus somewhat predictive of title performance, they are unlikely in and of themselves to drive or undermine the success of a game. We note this, in part, because of persistent rumors that some game developers have been jawboning game reviewers into giving their games higher critical review scores. We believe the publishers are better served by spending their time on the development process than by 'grade-grubbing' after the fact."
This discussion has been archived. No new comments can be posted.

Review Scores the "Least Important Factor" When Buying Games

Comments Filter:
  • by thijsh (910751) on Thursday November 26, 2009 @06:41AM (#30235480) Journal
    maybe it's the other way around... You only have to buy a sucky game *once* based on a raving review to *never* trust those reviews again. While your friends can comment properly on the game without some obscure metric like '8/10 overall'.
  • by DCFC (933633) on Thursday November 26, 2009 @06:42AM (#30235486)

    If a magazine or website is really scoring out of 10 or out of 100, then we ought to see some 1's and 2's.
    But we don't do we ?

    The researchers would find more utility in measuring the correlation between ad spend and score.

    Anyone think these two variables don't correlate strongly ?

  • Color me shocked (Score:2, Insightful)

    by Tar-Alcarin (1325441) on Thursday November 26, 2009 @06:57AM (#30235566)
    So, while there certainly is correlation between the review score and purchase numbers, there's very limited (if any) causation? With the immense integrity the game-review professionals command, who'da thunk it?
  • Trust (Score:5, Insightful)

    by Aceticon (140883) on Thursday November 26, 2009 @06:59AM (#30235582)

    It all boils down to trust (or more specifically lack of it) on the Game review sites.

    I'm not overly surprised that people don't base their buying decisions primarily on the review scores from game-sites. In most sites I've seen one or all of the following:

    • Creeping scores: games coming out now get higher average scores than games that came out 2 or 3 years ago
    • No mention of the bugs: a game might be full to the brim with bugs on release day and yet there is no mention of it or a passing reference to "the version we tested had some problems but this is a pre-release version and they should be solved before release" (not!)
    • Hype: game sites are the main culprits in creating/maintaining hype on often undeserving games. For example, before release Spore was being hyped to death by most game sites as a grand, revolutionary game - as it turns out, it wasn't even that much of a fun game and after release the hype-machine went suddenly quiet
    • Shallow reviews on just the beginning of the game: a lot of reviews sound like the person doing it just played the game for a couple of hours and then wrote the review. Plenty of games out there become pointless and boring after a couple of hours playing them, and yet that's often not mentioned in the game reviews
    • No mention of intrusive DRM: often enough the games come out with crazy phone-home, only works if Internet is always on, DVD-Writer-breaking DRM which installs a rootkit in our machine and yet not a peep about it in the main gaming sites. I suppose some might mention it if the activation process involved "chop your grandmother into little pieces and send them to the following address ..." but must would not

    Personally, I usually wait a while after the game is out and then go check user reviews. If your discount the "100%, great thing since sliced bread" ones (which come from fanboys) you'll usually be able to get a good picture of all the above mentioned points that the game sites miss (bugs, long-term (re)playability, intrusive DRM, hands-on-beyond-hype experience)

  • by Fallen Kell (165468) on Thursday November 26, 2009 @07:22AM (#30235694)
    Depending on how much the parent company spends in ads on the site/magazine, the score will be inflated higher. The few times that an editor or reviewer really did stand up and score a bad game as such, they were immediately fired by the advertising department. There was a big scandal a few years back when the editor actually spoke out against one such firing... I am too tired to look it up myself right now.

    Anyway, reviews anymore from the "gaming press" are total garbage due to this mechanic. The ads in the magazine are more important to the company than the reviews themselves. When was the last time you saw an EA game get a 1 out of 10.... And trust me, there are many deserving candidates, like the yearly sports rehash which change nothing in the game, just which player is on which team. Or Race Driver Grid, or Darkar 2009, or Rally Stars.... The magazines would just not post a review of a game when it gets bad because they don't want to potentially lose their ads from the publisher...
  • by RogueyWon (735973) * on Thursday November 26, 2009 @07:31AM (#30235746) Journal

    The problem is that you need a scale that encompasses everything from "hideously bad" to "sublimely good", and very, very few commercially released games these days actually fall into the former category. Sure, the usual anti-modern-gaming crowd here on slashdot likes to decry the latest overhyped blockbuster as "worst game ever", but in reality, pretty much every such game is "mediocre" at worst, and actually reasonably good fun if considered in isolation, on its own merits. It's not really fair to score a game down for being overhyped - only to review the game in front of you.

    Genuinely bad games with genuinely low review scores do exist. Even if you look at IGN, who are generally felt to "score high", you can use the review filters to find plenty of games with scores of 3.0/10 or less. These are mostly clustered on the PC, Wii, PS2 and handhelds - platforms with relatively low development costs prone to low-quality shovelware (which is by no means to decry all titles for those systems as low quality). However, the development costs for high-end games these days are such that you really can't afford to let an absolute stinker go out the door. This does make the odd rare exception that slips through, such as Lair, all the more deliciously awful.

    So yes, it's not a big conspiracy that you tend to get a clustering of review scores around the 7-9/10 mark. It's just a fair reflection of the overall quality of most modern big-budget games. Reader reviews, on the other hand, often tend to be callibrated to a less objective scale, and to take more account of factors such as the degree to which the game had been hyped (and to the kind of emotive factors that the console wars stir up), leading to a wider variation.

    You do, of course, get the occasional game where the "professional" review scores seem a bit out of whack. Modern Warfare 2 felt like a bit of an example of this to me; I could have seen it as an 8/10 kind of game, but I suspect that review scores above that are being hype driven.

    Ultimately, I find that the best way to use reviews isn't to go off meta-critic rankings or composite scores. It's to find a review site whose tastes generally accord with my own and use this as a rough guide. I already know in advance broadly which games I'm interested in. If I read the review, I use it as a guide-post and look for issues mentioned that are of particular importance to me. If a review flags that a game has an overly restrictive save-system, then I won't buy it even if the score is good, because I hate repeating content I've already passed unnecessarily. If a review criticises and marks down a game for not including online play, however, I won't let that deter me; it's not usually a huge issue for me, as aside from WoW, I'm primarily a singleplayer gamer.

  • by Shadow of Eternity (795165) on Thursday November 26, 2009 @07:35AM (#30235772)

    Yeah because $52.99 is such a big difference... I've never seen a used price from them that was even near $10 less than retail.

    Personally: Demos.

  • by Inda (580031) <> on Thursday November 26, 2009 @07:40AM (#30235808) Journal
    Does anyone buy games without trying them first? For me, if there is no demo, there is no sale.

    I haven't even read a review in 20 years.
  • by mwvdlee (775178) on Thursday November 26, 2009 @07:41AM (#30235810) Homepage

    OTOH, name me one single game that deserves a 1 or 2 out of a 100 score.
    Games may be bad, but their production value is rarely low enough to warant such low scores.
    If I were to publish a single picture of a maze, that would still be entertaining enough to score atleast a 3.
    Perhaps a virus would score a 1 or 2.

  • Demo's (Score:3, Insightful)

    by L4t3r4lu5 (1216702) on Thursday November 26, 2009 @07:44AM (#30235824)
    I buy games when I've enjoyed playing the demo. If there isn't a demo available, I don't buy it.

    Game world not designed to allow for demo-style play? Rubbish. You can sandbox an area of a GTA map, limit Dragon Age: Origins to one town, make level caps to prevent access to higher level play... It's just laziness.
  • Wait a few days (Score:3, Insightful)

    by DrXym (126579) on Thursday November 26, 2009 @08:35AM (#30236046)
    Most games are at best mediocre and at worst shovelware crap. It doesn't hurt to wait a day or two after release and gather consensus whether a game is worth purchasing or not. I don't get the urgency that some people attach to getting a game the minute it is released. If the game is THAT GOOD, then the reviews and consensus will bear that out, and if it doesn't, well you've just saved yourself a chunk of money.

    Be extra suspicious of games that embargo reviews, or allow just a handful of "exclusive" reviews to break the embargo. More often than not those reviews have been paid for in one way or another. Just like with other kinds of media there is usually a very good reason that publishers don't want you to know upfront what a game is like - because the product sucks.

  • by beatsme (1472991) on Thursday November 26, 2009 @08:53AM (#30236136)

    >But we don't do we ? There is a reason for that. It's a lot of hard work and cost bringing a product to market and generally, the real dogs are killed long before they hit the shelves. I've been reviewing hardware/software for 20 odd years now and I can only remember giving a score of less than 4 a handful of times. Equally, 9 & 10 is rare (for me). The vast majority of stuff is 'good enough' and merits 7 or 8 out of 10. TBH, I get really frustrated by constantly dishing out 7s and 8s and the few times something has turned up for review that's truly bad, I'm been delighted as it gives me a chance to have a real opinion.

    And this is exactly the mentality that has invalidated reviewer sites: you're looking out for the developers of these games, not the consumers who are your audience. Who cares if "it was a lot of hard work" when that hard work amounts to shit? If it's shit, say so, don't pretend its worthy because it was effortful shit.

  • by Moraelin (679338) on Thursday November 26, 2009 @09:18AM (#30236272) Journal

    Except you just illustrate the problem: something that's just "good enough" (which really just means "mediocre") gets an 8 out of 10. I'm sorry, but in a perfectly linear scale, "mediocre" would mean a 5. That's the kind of a number you could punch in a formula and get a correlation or anything else.

    Plus, if it were just a case of a honest review and the bad ones being already cancelled, the results would look much like the right half of a bell curve. You know, the curve with the below average ones removed. For virtually any sitze out there, it doesn't. It looks like a bell curve centered on 8 or 9, and which pretty much starts at 6 or 7. Sorry, that's not a case of the bad ones being already removed, that's a clear sign of an offset scale. It's what you get when the occasional "something that's truly bad" means you get to give a 5 or a 6, not a 1 or 2.

    And then there is the occasional reviewer whose curve looks like two spikes. The kind who churns 90% to 99% scores all year long, and then occasionally picks up some 10 year old freeware game so he can give _something_ a 5% score and fix his street cred. Or publishes a yearly smack-talking "top 10 worst games of all time" -- conveniently all 20 years old and from publishers which are no longer in business -- just to show that he's that unbiased and can give a low score too.

    But again, that's not being unbiased and fair at all, it's just trying to compensate one crap (or dishonest) job with another one skewed in the other direction. If it were a real fair and unbiased and non-skewed job, you'd get one bell curve centered in the right place, not two spikes centered near the extremes of the scale.

  • by LatencyKills (1213908) on Thursday November 26, 2009 @10:41AM (#30236838)
    I've been doing game reviews online for about a decade now, and as I look back over my reviews I find the ratings pretty much hit the full spectrum from 95% (Bioshock) all the way down to 14% (Dukes of Hazard - Racing for Home). I'm also a constant consumer of game reviews for games that I buy that I don't review. I think for all reviews that have a high degree of opinion (movies, books, videogames) it is important to find a couple of reviewers who feel like you do and stick with them. It's clear that specifically to the videogame realm there is a high degree of sellout (I won't name names) but here's a hint - avoid a review for a game that has a banner ad for that game on the same page as the review.
  • It's A-F (Score:4, Insightful)

    by Jim Hall (2985) on Thursday November 26, 2009 @10:41AM (#30236844) Homepage

    If a magazine or website is really scoring out of 10 or out of 100, then we ought to see some 1's and 2's. But we don't do we ?

    My wife and I were having this same discussion the other day. I was going through some reviews of games that just came out, comparing them to older games in the series. When I spotted one and mentioned the poor review, my wife asked what was the score. "6 out of 10". She was confused that a bad game got such a high score.

    I guess I've been reading these reviews for so long, I didn't think of it anymore. 10/10 is awesome, 9/10 is great, 8/10 is good, 7/10 is okay, 6/10 is poor, 5/10 and lower is terrible.

    "But when was the last time you saw a 5/10?" I honestly didn't know. Even the big-name movie tie-ins that we all know to be awful will somehow manage to score "6.5". I actually had to go look up some reviews to find lower than "6" - but they are out there. []

    I've started to view the "out of 10" or "out of 100" scores like the old A-F grading system we used in school. A is 9/10 or 10/10 ("A+"), B is 8/10, C is 7/10, D is 6/10 ... F is 5/10 or lower. It's not ideal to view games this way, but it makes sense of the review scores.

  • by Forge (2456) <> on Thursday November 26, 2009 @01:14PM (#30238070) Homepage Journal
    BTW: Neglected to mention. Editors choice awards are given out after consultation with Accounts receivables and advertising sales.

    I.e. If you want to win one, just buy lots of advertising.

You are in a maze of little twisting passages, all different.