7289618
story
Comments: 169 +- Games: Review Scores the "Least Important Factor" When Buying Games on Thursday November 26, @05:29AM
Posted
by
Soulskill
on Thursday November 26, @05:29AM
from the arbitrary-numbers-are-arbitrary dept.
from the arbitrary-numbers-are-arbitrary dept.
background: url(//a.fsdn.com/sd/topics/topicgames.gif); width:68px; height:77px;
games
A recent report from a games industry analyst suggests that among a number of factors leading to the purchase of a video game — such as price, graphics and word of mouth — the game's aggregated review score is the least important measure. Analyst Doug Creutz said, "We believe that while Metacritic scores may be correlated to game quality and word of mouth, and thus somewhat predictive of title performance, they are unlikely in and of themselves to drive or undermine the success of a game. We note this, in part, because of persistent rumors that some game developers have been jawboning game reviewers into giving their games higher critical review scores. We believe the publishers are better served by spending their time on the development process than by 'grade-grubbing' after the fact."
Executive ability is prominent in your make-up.
All trademarks and copyrights on this page are owned by their respective owners. Comments are owned by the Poster. The Rest © 1997-2010 Geeknet, Inc.
Let me be the first to say... (Score:5, Insightful)
Re:Let me be the first to say... (Score:5, Interesting)
There was the Mag Innovision letter to the editor after it's 17" monitor received the worst ranking in a roundup of 17" monitors. The Editor's choice award went to a Gateway 2000 monitor. The point of the complaint letter? "This is the same monitor, we just print different labels on the ones we ship to Gateway 2000." Or words to that effect.
So as a general rule I have very little use for published reviews of any product. Word of mouth, and personal trials work best. Also it's good to know what advertisers are allowed to lie about.
"This POS is the best on the market" -: Allowable lie.
"This overpriced crap is great value for money" -: Acceptable lie.
"This 500GB drive holds more data than 750GB of data without using compression" -: dangerous ground.
Parent
Re: (Score:3, Insightful)
I.e. If you want to win one, just buy lots of advertising.
Does anyone really believe the scores ? (Score:5, Insightful)
If a magazine or website is really scoring out of 10 or out of 100, then we ought to see some 1's and 2's.
But we don't do we ?
The researchers would find more utility in measuring the correlation between ad spend and score.
Anyone think these two variables don't correlate strongly ?
Re: (Score:2)
Part of the problem is everybody only quotes aggregates of the professional reviews. Take Farcry2 PC for example. Metacritic's Pro-review aggregate is 85/100 from 34 reviews. The user rating revew is 5.4/10 from ~580 reviews.
Re: (Score:3, Insightful)
Yeah because $52.99 is such a big difference... I've never seen a used price from them that was even near $10 less than retail.
Personally: Demos.
Re: (Score:2)
Re:Does anyone really believe the scores ? (Score:4, Informative)
Well, here in Portugal our only magazine dedicated to PC games (there's no market for more than one, really) gives plenty of low scores, and you're right, that's one of the reasons I trust their reviews. Besides the full text, the scores range from 0 to 100, and a few months ago they gave 4 to one game. Most games get between 50 to 60, but there are plenty of 30s and 40s.
I actually like the magazine very much, and had it subscribed a few years ago, but I end up playing only one or two AAA games each year, so it's not worth it. I hope they don't go under, though. We never had a strong PC games market, but now I fear it's reducing to new lows.
Parent
Re: (Score:2)
Ouch. What game got a 4?
Re: (Score:2)
Not sure, and I don't remember the exact issue, but it was some random RPG which featured 800x600 maximum resolution, isometric view, and it couldn't even smooth scroll; besides the texts had plenty of grammatical errors and the sound quality was worse than phone line. Oh, and plenty of story inconsistency.
Re:Does anyone really believe the scores ? (Score:5, Interesting)
There is a reason for that. It's a lot of hard work and cost bringing a product to market and generally, the real dogs are killed long before they hit the shelves. I've been reviewing hardware/software for 20 odd years now and I can only remember giving a score of less than 4 a handful of times. Equally, 9 & 10 is rare (for me). The vast majority of stuff is 'good enough' and merits 7 or 8 out of 10. TBH, I get really frustrated by constantly dishing out 7s and 8s and the few times something has turned up for review that's truly bad, I'm been delighted as it gives me a chance to have a real opinion.
Parent
Re: (Score:2)
That or the games are shovelware. The publisher knows they are crap and sees no reason to send review copies out when they'd earn an abysmal score for their troubles. Better to just push it out there and hope that grannies and five year olds will be fooled into buying it.
Re: (Score:2)
Not even that. It costs money to do the boxes, ship them out etc. They just can them and write it off.
Except you just illiustrate the problem (Score:5, Insightful)
Except you just illustrate the problem: something that's just "good enough" (which really just means "mediocre") gets an 8 out of 10. I'm sorry, but in a perfectly linear scale, "mediocre" would mean a 5. That's the kind of a number you could punch in a formula and get a correlation or anything else.
Plus, if it were just a case of a honest review and the bad ones being already cancelled, the results would look much like the right half of a bell curve. You know, the curve with the below average ones removed. For virtually any sitze out there, it doesn't. It looks like a bell curve centered on 8 or 9, and which pretty much starts at 6 or 7. Sorry, that's not a case of the bad ones being already removed, that's a clear sign of an offset scale. It's what you get when the occasional "something that's truly bad" means you get to give a 5 or a 6, not a 1 or 2.
And then there is the occasional reviewer whose curve looks like two spikes. The kind who churns 90% to 99% scores all year long, and then occasionally picks up some 10 year old freeware game so he can give _something_ a 5% score and fix his street cred. Or publishes a yearly smack-talking "top 10 worst games of all time" -- conveniently all 20 years old and from publishers which are no longer in business -- just to show that he's that unbiased and can give a low score too.
But again, that's not being unbiased and fair at all, it's just trying to compensate one crap (or dishonest) job with another one skewed in the other direction. If it were a real fair and unbiased and non-skewed job, you'd get one bell curve centered in the right place, not two spikes centered near the extremes of the scale.
Parent
Re: (Score:2)
As a tester who works with rating scales, I have to point out that a scale that has values that are never used is a pointless scale. If the range of scores reported by raters is from 5-10, then you don't have a 10-point scale; you have a 6-point scale. Also, if you're only using a few bands on the scale, you need to decide whether the raters need to be trained to discriminate more bands of the scale, or if your scale needs to be rewritten to allow such discriminations to occur, or if such discriminations ca
Re:Does anyone really believe the scores ? (Score:5, Insightful)
Parent
Re:Does anyone really believe the scores ? (Score:4, Interesting)
I find that when the spread is large, the extreme ends tend to point to piss-poor reviews that I can safely ignore. If there's a lot in either end, however, some of those are probably worth paying attention to. Most of the time, I look at a few "trusted" sites in the middle of the pack, such as IGN and GameSpot. GS invariably gives a lower score than IGN to any game, so I end up looking at that most of the time.
Finally, if a game gets mostly high scores, the low score reviews tend to be informative, and vice-versa.
So any time I buy a game or browse around a store, Metacritic is the first place I check, and then combine that information with the price of the game to arrive at a decision. So far the only times I've missed is when I didn't check Metacritic carefully.
Parent
Re: (Score:3, Interesting)
You misread him. His argument was that after that effort, the ones that deserve 6 and below are cut and don't reach reviewers, and therefore don't exist.
My personal opinion is that this means that he should rescale 6 down to 1, and leave 0 for what he now calls 0-5. If something is that incredibly rare, it doesn't deserve half the namespace to be allocated to differentiating between just how exceptionally bad it is.
But he didn't say anything about sympathies to the devs.
Re: (Score:2)
Re: (Score:2)
Why is FIFA 2010 with update graphics worth less then 9/10? Sure, if you already have FIFA 2009 it might be worth 4/10, but if it's your first ever FIFA game, it's probably worth 9/10.
Not saying I actually ever played the game or care about soccer games the slightest, just that the fact that it's an update doesn't mean it can't be reviewed for it's own merits.
Re: (Score:2)
Re:Does anyone really believe the scores ? (Score:5, Insightful)
The problem is that you need a scale that encompasses everything from "hideously bad" to "sublimely good", and very, very few commercially released games these days actually fall into the former category. Sure, the usual anti-modern-gaming crowd here on slashdot likes to decry the latest overhyped blockbuster as "worst game ever", but in reality, pretty much every such game is "mediocre" at worst, and actually reasonably good fun if considered in isolation, on its own merits. It's not really fair to score a game down for being overhyped - only to review the game in front of you.
Genuinely bad games with genuinely low review scores do exist. Even if you look at IGN, who are generally felt to "score high", you can use the review filters to find plenty of games with scores of 3.0/10 or less. These are mostly clustered on the PC, Wii, PS2 and handhelds - platforms with relatively low development costs prone to low-quality shovelware (which is by no means to decry all titles for those systems as low quality). However, the development costs for high-end games these days are such that you really can't afford to let an absolute stinker go out the door. This does make the odd rare exception that slips through, such as Lair, all the more deliciously awful.
So yes, it's not a big conspiracy that you tend to get a clustering of review scores around the 7-9/10 mark. It's just a fair reflection of the overall quality of most modern big-budget games. Reader reviews, on the other hand, often tend to be callibrated to a less objective scale, and to take more account of factors such as the degree to which the game had been hyped (and to the kind of emotive factors that the console wars stir up), leading to a wider variation.
You do, of course, get the occasional game where the "professional" review scores seem a bit out of whack. Modern Warfare 2 felt like a bit of an example of this to me; I could have seen it as an 8/10 kind of game, but I suspect that review scores above that are being hype driven.
Ultimately, I find that the best way to use reviews isn't to go off meta-critic rankings or composite scores. It's to find a review site whose tastes generally accord with my own and use this as a rough guide. I already know in advance broadly which games I'm interested in. If I read the review, I use it as a guide-post and look for issues mentioned that are of particular importance to me. If a review flags that a game has an overly restrictive save-system, then I won't buy it even if the score is good, because I hate repeating content I've already passed unnecessarily. If a review criticises and marks down a game for not including online play, however, I won't let that deter me; it's not usually a huge issue for me, as aside from WoW, I'm primarily a singleplayer gamer.
Parent
Re: (Score:2)
The French PC games magazine Joystick, apart from fairly objectives reviews, has a kinda effective and clear system. Apart from the regular notations (0-5 stars "a blank CD is marginally more fun"... to "for genre fans only" ... "Good Game !"), and they do use the whole gamut of grades, they add
- a Megastar status which means that the game pretty much is a must-have. They give out a handful of those annually
- side warnings if the translation sucks, or if the game requires a very powerful config to run, or i
Re: (Score:2)
And yet I've played plenty of games which I would recognise as over-hyped, but which I have also enjoyed. I mention Modern Warfare 2 in my post. Is it overhyped? Yes, wildly so. It's not even as good as its own prequel, due to a ludicrous plot which really shatters any suspension of disbelief. However, it's a lot of fun to play; the combat feels slick and precise, the weapons are well implemented and the scenarios provide a good degree of variety. So it's a very good game - but not one that's going to redef
Re:Does anyone really believe the scores ? (Score:5, Insightful)
I haven't even read a review in 20 years.
Parent
Re: (Score:2, Insightful)
OTOH, name me one single game that deserves a 1 or 2 out of a 100 score.
Games may be bad, but their production value is rarely low enough to warant such low scores.
If I were to publish a single picture of a maze, that would still be entertaining enough to score atleast a 3.
Perhaps a virus would score a 1 or 2.
Re: (Score:2)
Duke Nukem Forever?
It's A-F (Score:4, Insightful)
If a magazine or website is really scoring out of 10 or out of 100, then we ought to see some 1's and 2's. But we don't do we ?
My wife and I were having this same discussion the other day. I was going through some reviews of games that just came out, comparing them to older games in the series. When I spotted one and mentioned the poor review, my wife asked what was the score. "6 out of 10". She was confused that a bad game got such a high score.
I guess I've been reading these reviews for so long, I didn't think of it anymore. 10/10 is awesome, 9/10 is great, 8/10 is good, 7/10 is okay, 6/10 is poor, 5/10 and lower is terrible.
"But when was the last time you saw a 5/10?" I honestly didn't know. Even the big-name movie tie-ins that we all know to be awful will somehow manage to score "6.5". I actually had to go look up some reviews to find lower than "6" - but they are out there. [gamespot.com]
I've started to view the "out of 10" or "out of 100" scores like the old A-F grading system we used in school. A is 9/10 or 10/10 ("A+"), B is 8/10, C is 7/10, D is 6/10 ... F is 5/10 or lower. It's not ideal to view games this way, but it makes sense of the review scores.
Parent
Why (Score:2, Interesting)
And why? Because the grade-grubbing means that as of about 10-15 years ago, reviews are nothing more than adverts, and ratings are nothing more than auctions to the highest bidder.
I've *never* bought any game because of a review. Not even back when they were a bit more honest (e.g. in the Spectrum days, it was very common to see sub-50% and even sub-10% scores of games, some of them were even immortalised in things like a "crap games collection"). Game preference is completely subjective and neither word
I look at Gamefaqs (Score:4, Interesting)
Color me shocked (Score:2, Insightful)
Trust (Score:5, Insightful)
It all boils down to trust (or more specifically lack of it) on the Game review sites.
I'm not overly surprised that people don't base their buying decisions primarily on the review scores from game-sites. In most sites I've seen one or all of the following:
Personally, I usually wait a while after the game is out and then go check user reviews. If your discount the "100%, great thing since sliced bread" ones (which come from fanboys) you'll usually be able to get a good picture of all the above mentioned points that the game sites miss (bugs, long-term (re)playability, intrusive DRM, hands-on-beyond-hype experience)
Re: (Score:2)
Of course they're better: they're newer
Duh.
Re: (Score:2)
Not really certain about how much scores are creeping. There are a lot fewer really shit games out there these days, plenty of games I don't like, and plenty of let downs, but really shocking crap either doesn't make it or doesn't get reviewed. I've also seen games with huge full page adverts get panned in the same magazine, though I haven't read one in a few years.
Game bugs are a funny thing. Sometimes they're really massive game crippling bugs which should have been caught, but a lot of the times they're
People SAY they are not influenced by reviews (Score:3, Interesting)
Luckily with PC games you can test them for free. (Score:3, Informative)
Having said that, I do realise that this applies less to console owners, who are in a more difficult position because they generally can't test games before purchasing it, meaning they will have to live with a lower signal-to-noise ratio. (But then, they were the ones who chose to invest in a closed platform.)
Anyway, I'm fairly happy that most games are available for free testing (I'm usually not really in a rush to get any particular game), because - looking back - I can't really say that I found very many games that would've been worth my money if I had bought them (not even when they were sold at half 'list' price).. For the last couple of years the list would pretty much be limited to Portal, EU3, World in Conflict, Vampire: Bloodlines and Civ4 (and Arkham Asylum was OK too, just not at the current prices). Not a very long list, I might note.
In all, I would suggest people don't get consoles, as too much bargaining power is taken away from you in getting one, and too many games just aren't worth wasting money on.
Re: (Score:2)
XBLArcade and PSN do have demos of games.
Unless by demo you mean, pirate.
Review Scores are all payola.... (Score:3, Insightful)
Anyway, reviews anymore from the "gaming press" are total garbage due to this mechanic. The ads in the magazine are more important to the company than the reviews themselves. When was the last time you saw an EA game get a 1 out of 10.... And trust me, there are many deserving candidates, like the yearly sports rehash which change nothing in the game, just which player is on which team. Or Race Driver Grid, or Darkar 2009, or Rally Stars.... The magazines would just not post a review of a game when it gets bad because they don't want to potentially lose their ads from the publisher...
Re:Review Scores are all payola.... (Score:4, Interesting)
There is possibly an argument that because some firms let you keep the kit (sometimes quite expensive kit) and others always want it sent back, that this could affect your scoring but I try hard not to fall into that trap. That said, I often request review items I actually have a need for and this can actively work against it if it doesn't do what I'd hoped.
Parent
Review - yes. Score - no. (Score:3, Interesting)
I have three tiers of deciding on purchase of the game.
1. I read the review about what the game contains. I thoroughly ignore any "positive personal thoughts" about the game as marketing fluff. The negative ones do add to the value of the review but aren't all that important. I just read what is the concept of the game, and whether it is anything original, with potential - a good idea. If the review talks loads about graphics and sound and development time and prior franchise, even in total superlatives, it means the game is junk. A reviewer would concentrate on the really good points if it had any.
2.I check some Internet fora to see what people complain about. If there is a number of complaints about the same thing, it may turn me away again. The thing being "awful execution of the wonderful idea" is one of possible choices.
3. Then I grab the game off a torrent. After I'm through with it, I look back at how it felt. The only deciding factor is "I enjoyed it". Yeah, I enjoyed Stalker: Clear Sky, despite hopeless story, dull ending and reuse of content. I enjoyed Oblivion despite being dumbed down to knees level of Morrowind.
If the game passes the three tiers of classification, I buy it.
Amazon (Score:3, Interesting)
Demo's (Score:3, Insightful)
Game world not designed to allow for demo-style play? Rubbish. You can sandbox an area of a GTA map, limit Dragon Age: Origins to one town, make level caps to prevent access to higher level play... It's just laziness.
Re: (Score:2)
It's not really all that simple. A sand boxed area of GTA wouldn't feel like the real GTA does, and more than half the beauty of Dragon Age is how deep and rich and real the world is, one town even the best town wouldn't give you that.
Add in the fact that in a lot of cases the demos, even with everything cut out of it, are as large as the full game, and you're sort of in a bit of a tough situation.
Bought reviews (Score:2)
a modest proposal (Score:3, Funny)
Since the publishers are so keen on getting 8/10 reviews, lets replace the stars with a scoring system that just gives them more of what they want.
A good game gets one (8/10).
A mediocre game gets 3: (8/10)(8/10)(8/10)
And a terrible game gets a whopping 10: (8/10)(8/10)(8/10)(8/10)(8/10)(8/10)(8/10)(8/10)(8/10)(8/10) ... the publishers get what they want, and anyone with a calculator to hand knows what we really mean [google.co.uk]
Garrison Keillor says... (Score:2)
Because at Lake Wobegone Software Publishing, all the secretaries are efficient, all the managers intelligent, and all the developers are above average.
--
BMO
Re: (Score:2)
That's a really, really old quote, bud. You better Google it to be certain, but just yesterday I heard that Lake Wobegone Software Publishing was under-water [wsj.com].
Wait a few days (Score:3, Insightful)
Be extra suspicious of games that embargo reviews, or allow just a handful of "exclusive" reviews to break the embargo. More often than not those reviews have been paid for in one way or another. Just like with other kinds of media there is usually a very good reason that publishers don't want you to know upfront what a game is like - because the product sucks.
I never read reviews as reviews. (Score:2)
I never read reviews as reviews.
A review to me contains two or three things: advertising for the game, some more details about the gameplay that might be missing from the full-page graphic laden ad or TV spot, and possible it might compare the game to relevant reference points (other games, other relevant media, etc.). If I want an opinion beyond those bits of factual information I will look elsewhere - within days of a game being release there will be many opinions out there to pick from. Admittedly you ha
Real reason here: (Score:2)
Only a minority of gamers read reviews, get fact before buy. Theres a big group of people that just buy a game based on the box. And this groups is probably the bigger.
I am tempted to say that very few people that buy games are gamers (!). I mean, gamers as people that have gaming as his hobby.