Become a fan of Slashdot on Facebook

 



Forgot your password?
typodupeerror
Compare cell phone plans using Wirefly's innovative plan comparison tool ×
Math PC Games (Games) Stats Games

Fixing Steam's User Rating Charts 93

lars_doucet writes: Steam's new search page lets you sort by "user rating," but the algorithm they're using is broken. For instance, a DLC pack with a single positive review appears above a major game with a 74% score and 15,000+ ratings.

The current "user rating" ranking system seems to divide everything into big semantic buckets ("Overwhelmingly Positive", "Positive", "Mixed", etc.), stack those in order, then sort each bucket's contents by the total number of reviews per game. Given that Steam reviews skew massively positive, (about half are "very positive" or higher), this is virtually indistinguishable from a standard "most popular" chart.

Luckily, there's a known solution to this problem — use statistical sampling to account for disparate numbers of user reviews, which gives "hidden gems" with statistically significant high positive ratings, but less popularity, a fighting chance against games that are already dominating the charts.
This discussion has been archived. No new comments can be posted.

Fixing Steam's User Rating Charts

Comments Filter:
  • by Karganeth ( 1017580 ) on Monday October 06, 2014 @10:32PM (#48079745)
    http://www.evanmiller.org/how-... [evanmiller.org] This was linked before on slasdot IIRC. This is basically a blog post saying "hey this can be used on steam too, not just amazon!"
  • Valve Time (Score:5, Insightful)

    by UnknownSoldier ( 67820 ) on Monday October 06, 2014 @10:33PM (#48079751)

    Like all things Valve does on Valve Time, Steam is _slowly_ getting better so I'd imagine this will get fixed ... eventually.

    At least we can give a thumbs up or down to games. The ability to write reviews takes advantage of the best kind of marketing:

    Word of Mouth.

    • Re:Valve Time (Score:4, Interesting)

      by Nemyst ( 1383049 ) on Monday October 06, 2014 @10:56PM (#48079859) Homepage
      Valve have not shown a particular tendency towards using algorithms to fix things though - they generally just throw the work to their users, from tagging to rating to curating. While I agree that a fairer system to sum up reviews would be good for the consumer and good for smaller developers, I don't think it's what Valve are looking for. It's quite likely that showing more of the top sellers sells more copies of those games, which have dramatically higher sales volumes and prices than indies. That's a bigger cut for Valve.

      As much as people forget, Valve are not in this to make gaming a better place. They're there to make gobs of money, and have been rather successful at doing so thus far. Considering the sort of talent they hire and have hired in the past, if they truly wanted to fix things, they'd be fixed. If they're not, they either don't consider it important or have a reason for not fixing it.
      • Re:Valve Time (Score:5, Insightful)

        by Anonymous Coward on Tuesday October 07, 2014 @05:58AM (#48081331)

        As much as people forget, Valve are not in this to make gaming a better place. They're there to make gobs of money, and have been rather successful at doing so thus far.

        Valve makes gobs of money *because* they make gaming a better place.
        Valve is the sole reason why I'm probably not buying a console this generation, or maybe ever again. Gaming is so much better on PC these days that it just doesn't make sense to lock yourself into the console market anymore. And that's all on Valve.

        So you might be cynical and say Valve only cares about money, but the fact of the matter is that in order to generate that money they need a healthy market. Their interests are aligned with ours.

        Considering the sort of talent they hire and have hired in the past, if they truly wanted to fix things, they'd be fixed. If they're not, they either don't consider it important or have a reason for not fixing it.

        Maybe they've just got better things to do. Valve has their hands on so many pies that I'm sure they could double their workforce tomorrow and still keep everyone occupied.
        The new recommendations system is still new, I'm sure it's under close scrutiny and updates will be coming. But as someone else said, it'll happen on Valve Time.

        • by tlhIngan ( 30335 )

          Valve is the sole reason why I'm probably not buying a console this generation, or maybe ever again. Gaming is so much better on PC these days that it just doesn't make sense to lock yourself into the console market anymore. And that's all on Valve.

          And yet when the Xbone tried to do a Valve Steam (but with a few improvements) it was soundly killed off. I mean, the original Xbone inspiration was Steam - you could buy a game on disc, load it into your console and that's it. It was tied to your account, just l

          • Re:Valve Time (Score:4, Insightful)

            by Marc_Hawke ( 130338 ) on Tuesday October 07, 2014 @01:27PM (#48084567)

            Nobody liked Steam when it came out either. There were a lot of things that kept most people away from it:

            1. Always on. This was a problem both in internet connections (which were much more flaky back then) but also PC memory usage. Background processes were a gamer's worst nightmare before RAM sizes gained a few extra digits.

            2. "Vaulted Access." People still wanted physical copies. They didn't trust Steam to be around in 5 years and figured they wouldn't have access to their games anymore.

            3. Other things.

            So, Steam was ignored by a lot of people, except for the games that 'forced' them to use it (Valve games:...CounterStrike and HL2 mostly.) However, (and this is the magic Microsoft needs to find) Valve made steam not suck. People learned to trust it. "Yes" it will be available. "Yes" it will be convenient. "No" it won't hose your experience. And most of all..."Yes" it will be economical.

            Steam was considered draconian, until it proved not to be. And...importantly...it was 'optional' during that testing phase.

            • I boycotted Steam for the first few years when it came out. Precisely because it sucked for the reasons you said.

              10 years, 334 games, and many gamer friends later Steam works surprisingly well. EA has their piece of crap Origin, Microsoft aborted its Games For Windows Live; I think I'll stick with Steam.

            • by cyn1c77 ( 928549 )

              Steam was considered draconian, until it proved not to be. And...importantly...it was 'optional' during that testing phase.

              No, no, no... Steam still IS draconian; you've just have to deal with it and have gotten used to it.

              It's like the TSA-equivalent of gaming. It sucks, but is unavoidable, and eventually you just get used to periodically having your genitals fondled "for the good of the community."

              • by Agripa ( 139780 )

                It's like the TSA-equivalent of gaming. It sucks, but is unavoidable, and eventually you just get used to periodically having your genitals fondled "for the good of the community."

                Except that it is unlike the other TSA-equivalents of gaming produced by EA (*), Microsoft, and others, it is actually not aggravating to use and is without significant restrictions. Valve earned their goodwill and continues to do so. EA and Microsoft squandered it.

                And the Steam TSA-equivalent actually has some benefits like bei

    • Uhhh...what is there to fix really? Those "positive" reviews? Well you look closer and its stuff like "This is a bad game but its worth a dollar for the cheese factor" or "If you want to MST3K a game with your friends this is good for that". If you want real reviews go to Metacritic or YouTube, Steam reviews are filled with snarky PC gamers that can enjoy some "so bad its good" cheese.
      • by kubajz ( 964091 )
        As a matter of fact, does anyone know why Steam does not prominently feature Metacritic ratings anymore? Those really helped me choose games that I wanted...
        • by PJ6 ( 1151747 )

          As a matter of fact, does anyone know why Steam does not prominently feature Metacritic ratings anymore? Those really helped me choose games that I wanted...

          Maybe because games are given very high ratings [metacritic.com] that completely ignore the PC [metacritic.com], even when these ratings are supposed to be for the PC versions?

          I don't know about you, but when I see a AAA PC game also has a console version, I just stop right there and don't buy it, no matter what the ratings are.

          • by Agripa ( 139780 )

            I don't know about you, but when I see a AAA PC game also has a console version, I just stop right there and don't buy it, no matter what the ratings are.

            I give them enough benefit of my doubt long enough to watch some gameplay videos. Then I do not buy them.

      • Re:Valve Time (Score:5, Interesting)

        by Jupix ( 916634 ) on Tuesday October 07, 2014 @04:00AM (#48080943)

        What Steam reviews are actually filled with is information about the games... exactly what you should be interested in, as opposed to a score or a conclusion of some kind.

        The aggregate score in the style of "very positive" etc. can be useful in filtering out the genuinely terrible games, but outside of that, not so much. What's needed for decisionmaking is a lot of information, a search engine, and your own thinking. Steam provides descriptions, tags, and now reviews, and for me anyway it's been incredibly easy lately to figure out whether I want to buy a particular game, or at least investigate it closer elsewhere.

        Scores are almost completely worthless. Doesn't matter what kind they are (Metacritic, user review average, magazine review score). Steam has already done enough for the scoring system. What is there to fix? IMHO they should concentrate on important things like search, GUI and customer service, all of which are pretty terrible for 2014.

        • Re:Valve Time (Score:5, Informative)

          by hairyfeet ( 841228 ) <bassbeast1968 @ g m a i l . com> on Tuesday October 07, 2014 @05:20AM (#48081185) Journal

          Exactly. I have enjoyed some games that many considered truly terrible, "You Are Empty" springs to mind because while it wasn't anything new or revolutionary the Russian writers created a story that was as batshit as David Lynch movies and it had TWENTY FOOT TALL MUTANT ATTACK CHICKENS...seriously how could I not love a game that considered those a serious enemy type?

          But when I score a game like that I make it clear it is not getting a good review because its some amazing game, instead its strictly for the cheesy goodness. Take a game like "Two Worlds", sure the gameplay is just a generic "hack and slash" just like every other Diablo clone...but the dialog, that dialog was fricking great! It was like a fifth grader had all these "old English" words like "pray" and "forsooth" and had NO fucking idea what they meant but said "Yeah that sounds all cool and hamlet and shit, lets use 'em!" and when combined with actors that sound like they come from a dinner theater in SD doing Macbeth? Priceless!

          On the flipside you have GTA 4 which reviewers tripped over themselves to praise...Dafuq? That game is irritating as shit! Too damned many of the missions can only be gotten by kissing the correct ass and that involves hauling their stupid asses around and doing stupid mini-games to make their worthless ass happy...Why in the fuck am I taking a dope dealer out for dinner like I'm trying to bang him? Just give me the damned mission and STFU! Hell it was so bad there are fricking parody songs [metacafe.com] making fun of it, yet what did the majority of critics give it? 9s and 10s!

          So give me real gamers ANY day of the week, I'd much rather hear "Its decent, just don't buy it for more than $5 because X/Y/Z makes it not worth more" than listen to the critics which seem more and more to just kiss the industry booty.

        • by Agripa ( 139780 )

          The aggregate score in the style of "very positive" etc. can be useful in filtering out the genuinely terrible games, but outside of that, not so much. What's needed for decisionmaking is a lot of information, a search engine, and your own thinking. Steam provides descriptions, tags, and now reviews, and for me anyway it's been incredibly easy lately to figure out whether I want to buy a particular game, or at least investigate it closer elsewhere.

          I usually at least check out games I see other people on my

  • Why are they suggesting skipping straight to this hot mess instead of using a simple and well tested algorithm?
    • Why are they suggesting skipping straight to this hot mess instead of using a simple and well tested algorithm?

      Nor sure I understand? The article explicitly argues for using this simple and well tested algorithm. [evanmiller.org]

    • Uncertainties (Score:4, Informative)

      by Roger W Moore ( 538166 ) on Tuesday October 07, 2014 @12:06AM (#48080159) Journal
      Actually the problem they are wrestling with here is one that has science has had to deal with for a long time: the uncertainty on a measurement. The star ratings are a measure of the popularity of a game so what you are really asking is "given the ratings it received which game is best?".

      Unfortunately with a finite statistical sample you always have some degree of uncertainty and, within this uncertainty your data does not provide any ranking at all: you simply do not know which game is best to any sensible degree of certainty. However while correct this would lead to really confusing rankings since to be fair you would need to randomize the order within the uncertainty of each game's score. This would be complex and confusing to users!

      Instead what they suggest is using a confidence level limit: what score can I be confident that 95% of people would rate the game higher than? We do this all the time in particle physics when we put limits on some new physics which we looked for an did not see. For example the precursor to the LHC, LEP had a result that it was 95% confident that the Higgs boson had a mass higher than 116 GeV/c2 (IIRC). There are better ways to do this than the method they quote but since this is just a game rating and not science it's a fine method to use.
      • I have thought about another similar way to approach from a book I read to approach the above problem for continuous random variables, but I am sure that it could be adopted for what amounts to a categorical* problem here. The approach was a Bayesian estimator that supposed a previous expected value. So in this case, Steam might assign an initial rating of 50% as the baseline mean. As more real samples came in, the estimator was shifted toward the real mean and away from the Baysian prior estimator. In basi
    • by hey! ( 33014 )

      Because this method returns a ranking that seems more intuitively "right"?

      It's worth asking why that should be. Think about a rating scale of one to four stars. What does *averaging* those ratings mean? Yes, I know the *formula* for computing an average, but being able to *compute* an average isn't the same thing being able to *interpret* that average. Why? Because a two star rated item isn't really "twice as good" as one star, a four star rating isn't really "four times as good" as a one star or "twic

  • It's too bad... (Score:5, Interesting)

    by Charliemopps ( 1157495 ) on Monday October 06, 2014 @11:22PM (#48079977)

    It's really too bad the way Valve has screwed the pooch with Steam over the last few years. They literally had The gaming platform for PC all locked up. There was a time where I was desperately hoping they'd have an IPO so I could invest. But they tried to make the store so user friendly to Game controllers... a use case that may very well never become popular... that it's almost useless. Now, the only reason I think they are still relevant is because no one has bothered to try and challenge them. But I think they are 1 clever startup away from losing their position for good.

    There are games on steam to this day, that I cannot find... even using Google searches with the site:steampowered.com modifier. I have to go to the damned games external website and use their link to get to the thing I want to buy. I want to buy it and Steams own search doesn't bring it up because their search algorithm is so broken. I try to browse games and it limits what I can browse to a few dozen. Yet, when I go back 2 days later, its the same few dozen... why doesn't it just show me game after game until I've seen them all? There are over 4000 games on Steam!!

    And you know... I know what people are going to reply to me with... "You didn't click X!" or "You moron, you have to go to the blah page!" or whatever... I'm sure it's entirely my fault for not knowing how to do it right. But let me tell you something... the biggest moron on the planet can walk into Walmart and leave with less money. That's the key to their success. You cannot enter a Walmart and avoid seeing something you need to buy today. You don't have enough money? No worries there's a god damned bank on premises to give you a loan! It's easy to find something you like, it's easy to get it to the register and its easy to get it to your car.

    Why Valve? Why is it so god damned hard to give you my money?!?! I can go on the google play store and spend money with one damned finger! My 4yr old spent $20 on Angry birds slingshots before my wife locked her phone. He couldn't even figure out how to launch a game from your damned app!

    • Re: (Score:3, Interesting)

      by Anonymous Coward

      This I entirely agree with. The funny part of their whole new Discovery design? 99% of that could have been resolved with an Amazon-like system where you could simply get a list of "other people bought this" on all their store pages for games and allow one to readily go to the store page from near every game-related page (art, forums, etc). Of course to do that well would require having multiple tabs and perhaps even a like/dislike system per user to help sort things down--and there's Netflix to inspire

    • Just out of curiosity, what is it you can't find? I agree that finding stuff within Steam is hard, but I've yet to have Google fail me. Is it a really common word, or something along those lines?

    • by Anonymous Coward

      As funny as it sounds, you just cannot force a company to make more money sometimes.

    • Valve is extremely lazy, and Steam has allowed that. They have by far the majority of digital PC game sales, and most PC sales are digital these days. So they make tons of money doing very little work. This has allowed them to do what they really like doing: Faffing about with random projects, not worrying about any kind of deliverables.

      Unfortunately, lacking any real competition, Steam has no reason to really get better. The only store I would say is actually better than Steam is GOG, but they have the pro

    • by Ghjnut ( 1843450 )
      I think you're exaggerating the severity of shortcomings Steam currently has. While you may have trouble finding specific games and disagree with their approach to bubbling suggestions to the top, I'm still overwhelmingly satisfied with what they do provide. -On-demand game -Support for a plethora of indie titles -Emphasis on multiplayer/friends and communities -Novel monetization approaches (DOTA) -Strong push for cross-platform support with Valve heading the charge -Investment in bridging the PC/console d
  • Or use something like the Condorcet method to put all the games in order from most to least liked, and then assign each game a percentile ranking based on its position on the list.

  • by Anonymous Coward on Monday October 06, 2014 @11:41PM (#48080065)

    As a developer with a title on Steam, I can say ratings are just 1 of many issues with the store.

    I can't tell you how many support tickets I have to go through because steam can't properly launch or update the main binaries that are covered under their drm. Most of the time it's AV issues, but once in a blue moon it's steam borkin' out. (0.25-0.5% of sales)

    In the backend specific game information can't be shown to specific users. So I'm locked out of my real time statistics because my publisher has other titles from other developers...

    But the worst, the god awful worst, are games on Early Access that are pure shit or have been abandoned... They drag the entire system down and majorly screw over legitimate titles that are in Early Access. IMO Steam should have a purging system for these titles. Perhaps even give coupon codes to users who bought games that have been purged.

    As for the rating system. It definitely needs to be weighted. But there should also be incentive to give a ratings (even if they give a review anything) and there should be at least a "maybe" option. The thumbs up/thumbs down system doesn't really do it for me. Specially since I have negative ratings on my project such as "Game needs a German translation. 5/10" :\

    • by GNious ( 953874 )

      I can't tell you how many support tickets I have to go through because steam can't properly launch or update the main binaries that are covered under their drm. Most of the time it's AV issues, but once in a blue moon it's steam borkin' out. (0.25-0.5% of sales)

      I've concluded Steam is one of the worst pieces of software on my computer:
      * Ca 20% of the time it fails to start a game, or stop it, not recognizing whether the game is already running
      * It occasionally refuses to close, claiming some game is still running (it isn't), and can keep the system from rebooting
      * It constantly nags about changing a setting on my computer, despite being denied numerous times
      * it autostarts, despite never having been allowed to do so, and spams me with ads for games I cannot run on

  • by PacoSuarez ( 530275 ) on Monday October 06, 2014 @11:57PM (#48080117)

    If the only two choices are positive/negative (or thumbs up/thumbs down or some other equivalent 0/1 scheme), here's a formula that should work fairly well:

    (n_positive + 1) / (n_positive + n_negative + 2)

    So a single positive review gives you a score of .6667, and a single negative review gives you .3333. For large numbers of reviews, the score quickly converges to the actual fraction. If you don't have any reviews, you are at .5000.

    The mathematical justification for this formula is that if you try to use a Bayesian approach to estimating the true probability of getting a positive review, and you start with a flat prior, this formula gives you the average of the posterior probability after observing the given number of positive and negative reviews. The full posterior distribution is a beta distribution with parameters alpha=n_positive+1 and beta=n_negative+1.

    This formula is often used when applying Monte Carlo techniques to the game of go. I believe a lot of programmers simply start the counters of wins and losses at 1 to avoid corner cases (like division by 0), and they accidentally use the correct formula.

    • by Anonymous Coward

      here's a formula that should work fairly well

      It should, but it doesn't. It converges too quickly to allow useful comparison between things with 100 votes and things with 50k+ votes.

      Think about what happens if there are a 98 upvotes and 2 downvotes; your method rates it as 99 / 102 = 97.06%; meanwhile the error of the mean is still a whopping 3.75%. Ooops. Not good.

      TFA's suggests using Wilson's score. Using 95% confidence, Wilson's score is 93.0% for the 98 up : 2 down case.

      • You're using the lower bound of Wilson score. But the Wilson score is an interval, in this case 93.6% - 98.9%. The center of that interval is 96.2% which is pretty close to this algorithms 97,1%.

        I agree that you probably should use the lower bound of the Wilson score for these kinds of ratings, but you can't claim he's algorithm is badly wrong.

  • Discretionary XKCD (Score:4, Insightful)

    by Blaskowicz ( 634489 ) on Tuesday October 07, 2014 @03:02AM (#48080731)

    Yep there's one [xkcd.com] about it!
    It made me not get very enthusiastic about app stores and such.

  • by RogueyWon ( 735973 ) on Tuesday October 07, 2014 @04:49AM (#48081103) Journal

    I've noticed one big difference between the "professional" reviews on major sites and user reviews on Steam/Amazon etc.

    By and large, the professional game reviews tend to cluster their scores in the 6/10 - 8/10 range. You have to be exceptionally good to get above that level or exceptionally bad to fall below it. You also - in most cases - get relatively little variation between professional review scores. A game might be 8/10 on one site and 9/10 on another, but it is rare to see a gap larger than 2 or at most 3 points. It does happen - Alien Isolation has had professional reviews ranging from 4/10 to 10/10 - but generally only with unusual games that go outside the usual templates (like Alien Isolation).

    User reviews on the other hand, tend to be much more polarised. It's by no means unusual for games to pick up 10/10s from some users and 1/10s for another. Personal biases are much more likely to feature in user reviews ("I'm giving this game a 1/10 because I don't like something the developer said on twitter" or "I'm giving this game a 10/10 because I've spent the last 2 years boring everybody rigid about how good it is going to be and don't want to backtrack"). Often, the scores tend to average out in more or less the same place as the professional reviews once you have enough of both, but with much more divergence on the user reviews.

    So which is more useful?

    By and large - and with some important caveats - I find the professional reviews more honest and useful. A lot of people complain about the clustering of scores in the 6/10 to 8/10 range, but the nature of the modern games industry (quite risk-averse, with a lot of project oversight) means that most commercially produced games tend to fall into that range. If you assume a 6/10 is "not great, but overall more good than bad" and an 8/10 is "high enjoyable but not ground-breaking", then you're left with a spectrum into which most major releases fit. The industry does throw out the occasional piece of brilliance - which is usually recognised. And sometimes, things go wrong and it throws out the odd turkey (Aliens: Colonial Marines being perhaps the most recent example). When those things happen, most of the big review sites do seem to reflect them.

    But those caveats I mentioned before are important. The first is that at the end of the day, the people doing the professional reviews are still human and they still have their own biases, preconceptions and agendas. True, they have people watching them to make sure that they don't give free reign to those... but occasionally, those checks and balances fail. In fact, most of the big review sites have a few known quirks that you learn to watch for. Eurogamer, for instance (which despite the criticism I'm about to hand out, I do, in general, rate highly), has a real Nintendo-nostalgia fetish and a habit of over-scoring first party Nintendo games. At the same time, until fairly recently, it went through a phase of trying to shoehorn political correctness into its reviews and marking down a few games which committed real or perceived transgressions (though I've noticed less of this recently).

    The next big caveat with professional reviews is around bugs. The big review sites are often given pre-release copies of games, so that the reviews can go live before release. Indeed, a lack of pre-release reviews is often an early sign that a game will be a turkey (again... Aliens: Colonial Marines had a review embargo until its release day). Thing is, sometimes those review copies are unfinished code. And sometimes they aren't. But regardless, there is a tendancy for professional reviewers to either ignore or to be instructed to ignore bugs, on the basis that "they'll be fixed for release". And, surprise surprise, they often aren't fixed for release. User reviews are often your first warning that a game is a buggy mess - though on PC you do have to try to separate out the inevitable complaints that pop up on every new release's forms to the effect that "it won't run on my 8 year old PC running a

    • What we really need is something like Pandoras system. Something where my own ratings, as a user, are factored in. If 10,000 random users rank a game as 9/10 but I thought it was a 3/10, we obviously disagree on some things. Match me against people who review similarly to me if you want to help me find games to buy. Use the statistical algorithm as a fallback.
    • I agree here, the "pros" are very much pro-game, no matter what it is. I've seen reviews that point out all sort of flaws and it still gets 7 out of 10. A recent game full of hype, all reviews glowing, probably get game of the year, yet I look at one game play video and I can tell it's crap and dishonors the source material. But by the professional reviews it looks to be the perfect game for any gamer at all and should even be purchased by non gamers.

      Users reviews however will often bring up stuff the pr

  • If instead of talking about Steam, we were talking about iTunes Store or Google Play or XBox Live, 100% of the Steam users here would immediately start laughing about how stupid "those people" are, to be using the store to determine what to buy. That is obviously the very last intell source that you'd use. THAT WOULD BE STUPID.

    But somehow, if you're a Steam user, all your common sense happen to be inapplicable, whenever we happen to be talking about Steam (and you get your common sense back whenever you

There is hardly a thing in the world that some man can not make a little worse and sell a little cheaper.

Working...