Catch up on stories from the past week (and beyond) at the Slashdot story archive

 


Forgot your password?
Close
typodupeerror
×
The Media Entertainment Games

MetaFuture Talks Review Inflation 42

Posted by Zonk from the making-them-feel-better dept.
MetaFuture, a game journalism analysis site, has recently refocused on review scores from the big gaming sites. The author takes an interesting approach, taking a look at Gamespot's review spread and IGN's tendencies. Unsurprisingly, both sites tend towards the 7 to 9 range, making it debatable whether their numbers are actually useful. The site's eventual goal is to normalize the review scores from the major sites, and actually make them useful. From the article: "Games will still get an average score from all contributing reviews. But a site's contribution to that average will depend on that site's own individual normal curve-- with the immediate left and right of the bell's tip signifying three stars on a scale of one to five. Watch the drama as the biggest sweethearts see their 8.4 score for Gun and Car IV get pegged as three stars." This is the reason Slashdot videogame reviews don't have numbers anymore.
This discussion has been archived. No new comments can be posted.

MetaFuture Talks Review Inflation More

MetaFuture Talks Review Inflation

Comments Filter:

  • Selection effects? (Score:5, Insightful)

    by JackBuckley ( 696547 ) on Tuesday August 08, 2006 @09:48AM (#15865505) Homepage
    I agree that fanboy magazine ratings should be viewed with scepticism, but I also worry about "normalizing" the reviews. There is an underlying assumption here that the population of games is symmetric in the distribution of quality. This may not be true, if, for example, games which are in the low quality tail are not released (game companies are strategic actors, right?) at a higher rate than games in the high quality side of the distribution. In addition, gaming magazines do not necessarily choose which games to review at random either--they either review interesting games with a higher probability of being of high quality (if you want to be kind to the industry) or else review in response to payola/swag (in which case it is the companies strategy which matters again.)

    So, the question is, conditional on nonrandom selection of games to release and nonrandom selection of games to review, what should we expect the distribution of quality to look like? My guess is that this distribution is nonormal and is skewed with more observations in the higher quality tail. This does not necessarily mean, however, that the reviews are "fair," but it suggests that the question is more complicated than a simple "grade inflation" argument.

    Note that I am also making an assumption that quality is judged in some sort of absolute terms, and not relative to the other games that are released. There are probably some other assumptions lurking in there as well. Just my $.02

    • In addition, game reviewers tend to review the types of games they enjoy playing. If a review company has a guy on staff that really likes fighting games, they'll likely give him the fighting games to review. This kind of thing likely tends to skew the results as well depending on the quality compared to other games within the same type.
      • Isn't that a good thing? someone who doesn't give a shit about fighting games probably isn't going to notice the little subtleties of a new title. You want someone who knows all the various fighting games inside and out so he has a basis for comparison. If I'm out to buy a fighting game I want a reviewer to say "this game is good, but tekken is better". Then I know which fighting game is better. If the reviewer says "this game sucks, Civilization 4 is much better", how does that help me if I'm trying to dec

        • Re:Selection effects? (Score:3, Insightful)

          by iocat ( 572367 )
          Since I actually read reviews, and don't just look at the scores, I know I vastly prefer to read a review by a fan on the genre (who might be more forgiving on the score) than someone who knows nothing about it, regardless of any grad inflation that may go on.

          The great examples are NASCAR and wrestling games. The typical NASCAR game review is like "well, if you like turning left a lot and you have a lobotomy, you may enjoy this... 3/5." Contrast that with Ivan Sulic's exceptionally well written review [ign.com] o

          • "Additionally, this notion that the "average" game must score 5 on a 10 point scale is retarded. If you have a 10 question quiz, your average person will probably get 7 right. That's why a 70% is usually a C (average). IGN, GameSpot, Game Informater, etc's scoring system may trend towards scoring 7 - 9, but that's simply in line with the way most people grade things on a 10 point scale. On a 5 point scale, you see a lot more threes, but so what? A 3 doesn't tell you any more than a 7 -- ultimately you need
            • Maybe if you tallied random school grades you'd get 70% as average, but that's because the quizzes are created to hit that average on purpose.

              Yep, that's all I meant. Your average student will score 70%, becuase the tests are designed for average students to score 70%. So 70% for a game is something that most people can instantly identity with. Most games today, when you come down to it, represent a reasonable ROI for your entertainment dollar (the baseline quality has gone up dramatically in the past 1

        • well obviously you should buy Civilization 4 because it is much better.

          Pardon the sarcasm because I totally agree with you on that. Though, for that reason I think reviews should be done by more than one person. Let's say a team of 5 people review a fighting game, I'd say it should consist of: a Fighting gamer, an FPSer, an RTSer, an RPGer, and a non-gamer (sorry if I didn't include other genres) If all of them looked at the game from their prospective angles, I bet you'd get an amazingly indepth view at t

          • The problem with that is that no one ends up completely satisfied. Take warcraft III. I like RTS games and was expecting it to belike starcraft except with orcs and humans instead of zergs and terrans. But instead I got something halfway between starcraft and diablo II. I like starcraft and I like Diablo, but warcraft III didn't work. I could only level up my character two or 3 times so it didn't make a good RPG, but my armies were limited to a couple dozen soldiers so it wasn't much of a RTS game.

            A gam

            • true, warcraft 3 didn't work out for me either. But say it was implemented in such a way that you could level your hero many more times and outfit with many more abilities and equipment, or maybe that every character levels, I'm sure it would have been more to my liking.
              The same thing happened to me and WoW, what I expected was the ability to play the game as a giant RTS (no, not in the way you think i mean :) I mean in such a way that, if the horde decided to amass an army to raze ironforge, then they coul

              • hmmm... way off topic now... But yeah that sort of thing is what I'd like to see from a massively multiplayer game too. As it is now WOW isn't much different from Diablo II only it allows bigger teams and meeting up with people is all in a 3d environment instead of in a irc-like room like in diablo. It's just teams of around 30 people playing in parallel, not massive armies having epic battles. A shame really. Maybe that's all current networking technology allows.

                I don't see why it couldn't end though. I

    • You are looking at it from the wrong perspective. The reviews should be based largely on other games that have been reviewed. A game that has never been reviewed by a given publication will not have a score attached to it (obviously) and thus cannot be compared to those that have been reviewed. You are asserting that the point of comparison should exist outside of lab evaluation. Games should be compared to those that have been reviewed with a much smaller bias toward a top score.
    • I think what reviewers say going in is that, "okay, 5 is average, if you get above 5, hey, you're doing something right." But then they hand out 6's and 7's and the companies are like "OMG!!!! totally unfair!!! That's a failing grade! And it's a good game!" (I think this actually came up in Electronic Gaming Monthly about 10+ years ago when they wanted to defend giving a game a 7 on just that basis - it *was* above average, but that's not "good enough" a reason to leave it at 7. Maybe the game was Supe

      • [reviewers] hand out 6's and 7's and the companies are like "OMG!!!! totally unfair!!! That's a failing grade!"

        For me, a "failing" review is a 3/10 or less, not 7/10. I would score a game 10/10 if it could not be improved in any way at all — perfect control scheme out of the box, perfect camera, perfect story and art, etc. Some novelty to a game justifies a 7, but anything that is "just another" should be a 6 at best, and usually a 4 (since "just another" games tend to have a goodly number of minor f

        • Yeah, but the article is talking about Gamespot and IGN and the like. Getting a 5 out of 10 from them is like them saying it's the worst game ever made, and a game store clerk is justified in stabbing you in the face if you pick up the box.
    • The percentage system is something of a mystery, but it does correlate fairly closely to how the US A-F system, translated to the 100 point scale, is supposed to be applied to grades in big US Universities.

      The center of the distribution is supposed to be around the high 70s/low 80s. (C+/B-).
      Back when I was the TA monkey handing out grades, the recommended distribution was 40% Cs (70s) 33% Bs (80s) 10% As (90s), and, well, the Fs (
      The distribution looks pretty much like that. Incidentally, the same system c
    • I don't know. What's a `strategic actor`?
    • Those are different though as they only average all the review scores for particular games. This site is attempting to 'rate the rating' and thus apply weights to their review scores when calculating it into the total score--thus bad reviewers who consistently give certain genres and whatever a high or low score won't inflate or deflate the overall rating by as much.

  • Reviews are only useful when... (Score:5, Insightful)

    by mikeisme77 ( 938209 ) on Tuesday August 08, 2006 @09:50AM (#15865520) Homepage Journal
    Just looking at the end score in the review for ANYTHING is useless. The usefulness in a review is in reading the comments of the reviewer and understanding the reviewers preferences in games by looking at their reviews of other games you're interested in. The trick is to find a reviewer with similar thoughts on genres and such as your own, that way their review is relevant to you. The other trick is finding well thought out, well explained reviews--ones that tell you EXACTLY what the shortcomings and pros of the title were, this way you can decide if the shortcomings are shortcomings to you or if you just think the reviewer is being anal.

  • Who cares for hose numbers? (Score:4, Interesting)

    by Denial93 ( 773403 ) on Tuesday August 08, 2006 @09:54AM (#15865549)
    I gave up on game reviews when I worked at a game company and my boss explained his frustration with the review industry to me. He hated having to use dishonest means to catch the reviewers' attention - in particular, photoshopped preview game stills - but he said this was the only way to compete for those exceedingly rare "Game of the Month" etc. spots. He showed me a six page "exclusive preview" by a competitor and explained the only reason it was that big (as opposed to a small paragraph somewhere) was that the competitor had moved the preview session to Florida and paid the expenses of half a dozen editors for an extended weekend. How much sense does Florida make, when you're a European software developer and neither the game not the magazine are released in the English language? Now editors don't need to allow these tactics to work, but they evidently did. And worst of all, the magazine was the most respected (perceived fair) one in that market. Others were way more blatant about their corruption and deception of readers. The one that eventually made ours Game of the Month did so because we'd agreed to give considerable amounts of merchandise to readers who participated in a lottery.

    When I want opinion on a game, I turn to a friend with a good taste in games, a fat pipe, and lots of time.

  • I give this article an 8/10 (Score:4, Funny)

    by gEvil (beta) ( 945888 ) on Tuesday August 08, 2006 @09:58AM (#15865581)
    Zonk says: This is the reason Slashdot videogame reviews don't have numbers anymore.

    And here I thought it was because everyone was always giving you shit for rating everything 8/10...

  • Constant offset (Score:3, Interesting)

    by NialScorva ( 213763 ) on Tuesday August 08, 2006 @10:03AM (#15865609)
    A friend of mine is a reviewer for a gaming site that does their reviews on a [1,10] scale. Apparently the editors tell the writers to rate them on a [1,5] scale, then they just tack on a bonus 5 points to whatever the writers say.
  • There's already a site that does this. metacritics [metacritic.com]
    • Can I get a third differential? I'd like to know if the metacritics are capable of detecting a magazine that is trying to appear average by artificially lowering sucky games (that buy few ads anyway, ideally are also a small European company that's not got much else going on) in order to offset their high reviews of games that buy ads...
  • I've been finding game reviews increasingly irrelevant. I still like Edge's, but sites like Gamespot and IGN are generally pretty useless for figuring out whether a game is fun or not. I often go by the internet community's response: frequently random forum posters make far more insightful points than reviewers.

    • Re:Increasingly irrelevant (Score:1, Interesting)

      by Anonymous Coward
      Guess I'd be in the same boat as you then. The last reviews I actually found really useful were done about 10 years ago in 2 different magazines. EGM had an OK system where four different people (most of whom were recurring) would given their thoughts on a game. GamePlayers, I think (it's been awhile), used to do a scale from 1-100% in addition to some very long and in-depth articles. Last time I picked up EGM, the "reviews" were much shorter (about 1 paragraph) and far less diverse; GamePlayers has bee

  • I agree, but... (Score:4, Interesting)

    by tygerstripes ( 832644 ) on Tuesday August 08, 2006 @10:11AM (#15865671)
    I've always been of the opinion that games reviews are becoming increasingly inflated, but it's too easy to look at this in a sceptical "Fanboys have no sense of proportion" light.

    Bear in mind that the games industry is not static. Not only are the technologies and concepts used in games and development constantly evolving and improving, but the budgets and resources being thrown at them far outstrip those seen 20, 10, even 5 years ago.

    Inevitably this causes the leading edge of the games market to progress faster than our sense of cynicism and ennui, so we are more and more impressed with each new release. That's what ultimately gives a game a great score - how impressive is it? Scores for big releases will tend more and more towards the higher end (especially if we try to rate games comparitively with other fairly recent releases).

    The term review inflation is surprisingly accurate in that regard but, while steady economic inflation is not a problem, it is troublesome in reviews where there is a fixed range of possible scores. The dollar can become worth 1% of its original value and there's no issue - as long as it's in line with market growth and currency values in other economies - becuase its value is defined by the market-perception of its value and has no technical limits. The star-rating system in reviews is inherently fixed, so inflation is a bugger.

    Allowing the rating-system to inflate freely would get around this problem, but then you're looking at free-flux exchange-rates between reviewers and the issue of fitting 210 stars into a box-out and, frankly, the only sensible answer is for peeps to grow up and take everything they read with a pinch of salt. Hey, it's a valuable life-lesson. Learn to read, dammit.

    • That's what ultimately gives a game a great score - how impressive is it?

      While I think that is unfortunately probably true for the review score, the question the gamers want answered is "how fun is it?"

  • Part of the problem (Score:3, Insightful)

    by miyako ( 632510 ) <miyako AT gmail DOT com> on Tuesday August 08, 2006 @10:20AM (#15865751) Homepage Journal
    I think a big part of the problem that people have is that they simply don't know how to read game ratings properly. I know that, personally, I usually find that when I look at the ratings for a game, along with the text of the review, it can lead me to a fairly accurate understanding about how well I will like it.
    The first thing is that I think people think of the ratings system in terms of absolute "10 is a great game" "5 is mediocre" "1 is crap" scores. That's not really accurate. In general the score must be considered from within it's genre. A football game with a rating of 10 might be excellent for people who like football games, but I certainly wouldn't enjoy a football game with a 10 rating any more than a football game with a 1 rating, because I don't like football games. Likewise I might enjoy an RPG with a score of 7 or 8, but other people would find it tedious, because they don't like RPGs.
    The other problem is that I think people expect scores to fall in a fairly normal distribution. The problem is that game quality isn't a normal distribution. There are a lot of games that are made that people might not consider fun, but they are at least semi-playable. If you consider a game that might get a 1 or a 2 rating, it would have to be something with severe software flaws that kept the game from even being playable. On consoles at least, no matter how bad a game is, it's rare for a game to be so bad that a determined person couldn't play it (even if they didn't enjoy themselves.). If you look higher in the ratings, it's similar. Most games tend to be clustered in the 7-9. To understand why I think you need to really understand what the ratings in that range mean. When I'm looking at a review, and I see a game with a 10, that tells me that the game is well executed and should appeal to the majority of gamers even if they aren't particularly fans of the genre of the game. A 9 generally says that the game is on par with the best games of that genre, and introduces some new concepts to extend it. A rating of an 8 generally tells me that the game is solid and people who are fans of the genre will probably enjoy it, but it might not appeal to people who aren't specifically fans of the genre, or of the series. A game with an 8 might either have a few flaws that lower the overall experience, or it might be a solid game that fails to offer anything innovative. A rating of a 7 generally says that the game is weak. A 7 tells me that someone who was a big fan of the genre or series might enjoy the game, but that there are probably some flaws that other games in the series or genre have fixed, and that the game either has some fairly large flaws that non-fans won't be able to over look, or its very formulaic and will be boring to someone who isn't a huge fan of that forumla. Finally, looking in the mid range of 3 to 6, you generally see games that are lacking something that generally a game should have, but which doesn't render the game unplayable (when I say unplayable, I mean physically the game won't run, as opposed to the game having responsive or intuitive controlls). A 5 or 6 for example says that a game probably has some severe playability issues that interfere with enjoyment of the game, as well as having some bugs and lacking features that are standard for the genre. A 3 or 4 generally says that either because of various bugs or lack of features, there isn't much "game" to the game at all.
    What I think it boils down to is that games first of all need to be rated only within their own genre, because it's hard to set a single scale for games across different genres. Using a normalized scale seems intuitive, but it doesn't work because the quality of games isn't a normal distribution, instead it's skewed so that there are generally a lot of "Ok" and "so-so" games in the 7 to 8 range, a lot of games that lag behind because of various problems, and a very few gems that get the coveted 9's and 10's. It's also hard to quanticize fun. A reviewer can really only rate a game based on what he

  • Money for Reviews (Score:3, Informative)

    by Enderandrew ( 866215 ) <enderandrew&gmail,com> on Tuesday August 08, 2006 @10:34AM (#15865873) Homepage Journal
    Atari has openly paid for reviews in the past. Anyone remember the plethora of 9 and 9.9 scores for Driv3r? I am not bothered that most games rate a 7-9. I'd assume that most AAA titles should be on the good end of the scale. What bothers me is how many games receive a 9, 9.9 or 10. Shouldn't those be reserved for the truly exceptional? Or are there 10 games released every year that are TEH GREATEST EVA?
    • > What bothers me is how many games receive a 9, 9.9 or 10.
      > Shouldn't those be reserved for the truly exceptional?

      Yes. Go play "Sacrifice", the last game that honestly deserved that high a rating. (Black and White, I'm looking in your general direction!)

      Eh, now I'm in for it, a karma thrashing. Might as well go for broke: Linux and "sux", both end in "ux". Coincidence? Natalie Portman, nice lips, pretty face, body like a little boy. Uhh, Java, yeah, impressive to the natives. When you wake up

  • Should be more like film critics (Score:4, Insightful)

    by Bender0x7D1 ( 536254 ) on Tuesday August 08, 2006 @11:44AM (#15866608)
    People should rely more on reviews from people that have the same gaming opinions as they do instead of some number. Consider: How many poorly rated movies do well at the box office, and how many highly rated movies do poorly? A lot. Check out Yahoo! movies or similar site and compare the critics to the people. They are never the same and rarely similar. Why? Movie critics see a lot of movies, so are biased towards the storyline and acting instead of a big action sequence. So, they view movies differently than I do.

    Extending that to video games; a reviewer who enjoys FPS games is going to give a high rating to the latest shooter with great graphics. I like older FPS games, but hate the direction that the industry has gone with newer games. So, if a reviewer is a fan of the genre, and I'm not, should I use their review? Of course not! I hate RTS games, so even if one had a 10 rating I wouldn't buy it. However, maybe someone does something new and it is worth my time and money to give it a shot. How do I know? I need to find a reviewer who doesn't like RTS games and get their rating - if they give it a 7 or 8, but they don't like RTS, then I should look into it.

    So, how do you find these reviewers? Give ratings to the games you have played, maybe separated by genre, and then go looking for reviews that are close to your own and look at the name of the reviewers. Then search by reviewer to see how close their ratings are to your own, pick the closest (or some sort of combination - Alice for RTS and Bob for FPS). Now you have some reviewers you can trust will like the same games you do, and you can shop accordingly.
  • Do away with the 1-10,5 star, or 100% rankings I say (god that PC Gamer 100% scale is rediculous, what exactly is the difference between a 95% and a 96%, or even 90% for that matter?) The text is whats important, I say if the reviewer wants a quick capsule summary, make it a 3 point scale: "Buy it", "Rent it", "Avoid it".
  • All you people who are harping on about the quality of games not being normally distributed have a point, but you are missing a big point too in that while the quality of games may or may not be normally distributed, the quality of gaming review sites have no such restrictions. There are be some payola, pushiver review sources, other hard-nosed perfectionists, with the majority of game reviewers falling somewhere in the middle. Indeed, I felt likt that was the whole point of the article in claiming that th

Slashdot Top Deals

I've noticed several design suggestions in your code.

Close