Follow Slashdot blog updates by subscribing to our blog RSS feed

 



Forgot your password?
typodupeerror
×
Role Playing (Games)

Overcomplicated MMO Betas 64

Heartless writes "On the heels of Vanguard's beta 1 announcement, Heartless Gamer blog has an article looking into why MMO beta processes are overly involved and detracting from the game they are meant to improve. From the article: 'But why even have such a process in the first place? If they honestly think they are going to get any sort of actual *testing* (I use the term loosely) from an over-hyped MMORPG community... they obviously failed basic MMORPG sociology. I could link hundreds of beta leaks and broken NDA contracts, but what would be the point? What you need to know is the fact that betas are infiltrated by those that want sneak peaks at the game. Definitely not by those that truly wish to test the product. Internal testers and paid testers have proved for years to be able to produce very finished products in the single player market.'"
This discussion has been archived. No new comments can be posted.

Overcomplicated MMO Betas

Comments Filter:
  • Gimmicks (Score:3, Insightful)

    by danikar ( 896514 ) on Tuesday October 18, 2005 @06:39PM (#13821939) Homepage
    In beta tests the company puts in a few neat features that will attract players. They let people play it for free to stress test the servers, not look for bugs. And hopefully get some fanboys that got really excited about a few gimmicks to promote the game for them. lol, NDA even the company that put it out doesn't care that much about it. They know for a fact it will be broken. It is free marketing.
    • I remember hanging out the WOW beta boards back before Beta actually went live. They thought it would be a good idea to award "fansites" with 3 slots apeice to give away in addition to getting the random 1000 or so players. Bad Idea. Lots of fansites kept the 3 slots for themselves, and as far as Blizzard was concerned, tough noogies. The worst ones were the fansites who paid lip service to the idea of a giveaway, yet just gave the slots away to friends or hell, just lied about it.

      Sufficed to say, we all
    • Re:Gimmicks (Score:2, Informative)

      by Anonymous Coward
      yup what he said, you shouldnt even do an open beta test if your looking for bugs, you need to have your show stopper bugs nailed down before you ever let the public see your game.

      This has been the bane of many MMO's that have come and gone. Pushed to get the game out too soon, rushed into a buggy beta test that reveals nothing but how premature the
      game really is, it ruined Earth and Beyond for example, beta'd too early, too buggy, too unbalanced and for too damn long (linke 6 months or more of open beta,
  • Weight Of Numbers (Score:4, Insightful)

    by Anonymous Coward on Tuesday October 18, 2005 @06:43PM (#13821978)
    10,000 players...
    1% genuinely beta test
    That's a 100 person QA team - far bigger than the typical MMO will ever see.

    Now up those numbers to:
    50,000 players...
    1% genuinely beta test
    5-10% vocally bitch about every weird bug and quirk they find
    Now you're looking at 500 decent QA testers and another 2,500-5,000 pain in the ass guys who're maybe worth 1% of a tester each but cumulatively do still add up.

    A beta test doesn't have to have every player responsibly beta testing. Sheer numbers ensure the end effect still gets met.

    Besides, by public beta, the main thing that should be getting tested is load and the weird load quirks caused by 5,000 players all deciding to try the same exploit etc. That, whether they're good testers or bad, still happens. Arguably it happens even better if they're "bad" beta testers as they're more likely to do things they "shouldn't".
    • Re:Weight Of Numbers (Score:4, Interesting)

      by dhakbar ( 783117 ) on Wednesday October 19, 2005 @12:29AM (#13824094)
      As someone who has worked QA on a variety of SOE games and have had to read through player-submitted bug reports, there are FAR fewer than 1% of the player base submitting bugs in the first place, and the ones who bother cannot be considered "decent QA testers." Someone else mentioned it earlier... the beta phase is merely to hype the game. It's incredibly rare that players, even those with legitimate bugs, submit reports that are useful for QA purposes.
      • It doesn't help that in a lot of the games you're given a bug report window with a 200 character limit to describe the bug. Many times that is just not enough room to adequeately describe what the bug is.

        I know long reports are hard on the QA folks, but for a real issue it can be necessary.

        A lot of players submit bug reports on stuff they just don't understand too. It's part of the game, but they havn't realized it yet (my fireballs don't work on this door!).
    • by Moraelin ( 679338 ) on Wednesday October 19, 2005 @05:31AM (#13824950) Journal
      Quantity doesn't always equal quality. If that were the case, we'd still be using the old no-ranking search engines on the Internet, and Google's attempt at sorting out by relevance would have silently failed. At some point, you'd rather just get the actual info, and not scroll through 10 pages of crap before you find anything relevant. One more guy posting "my class sucks" threads is just more noise, not more signal.

      In other words, when I Google for something, I'd rather have 1 link that is exactly what I want, than 100,000,000,000 irrelevant links. The same goes for beta-testing, _if_ the goal is actually to beta-test, and not just to get some free publicity: I'd rather have just 50 people actually professionally looking for bugs, than 50,000 whining about everything else.

      Having 500 people who genuinely test for bugs, is _worthless_ if their signal is drowned in the noise from 50,000 people posting like there's no tomorrow about how your game sucks ass because his Priest doest't _start_ with the Mages' level 50 spell. (That's sadly not even a joke. Something Awful once had a parody of an open letter to Sony, in which they asked for really ludicrious stuff, including _literally_ that a level 1 priest should start with the most powerful mage spell. Much to their surprise, they got a helluva bunch of emails aggreeing wholeheartedly.) Or how it sucks ass and is unbalanced because it doesn't _force_ everyone else to group with his Priest that bought everything _except_ healing/buff spells. (Add a long circular-backpatting whine about how players are idiots and don't appreciate how useful that priest is with his mace alone.) Etc.

      And it goes downhill from there. The guy who discovered a bug and filed it, will start _one_ post. The guys arguing that their characters should have 100% resistance to damage and an insta-kill spell that costs no mana, will start one per day. And more often than not, spill into the other topics too. (Surely a post about how a mage spell sometimes fails with no explanation, not even a "your spell was interrupted" message, is _the_ right place to post about how either (A) you mages had it too good and it was about damn time that spell got a downgrade, or (B) about how we mages are the whipping boys of the devs, and they downgraded yet another of our spells. Doom, gloom, run for the hills, and all that.)

      Welcome to the wonderful world of looking for the proverbial needle in the haystack.
      • I'd rather have 1 link that is exactly what I want, than 100,000,000,000 irrelevant links. The same goes for beta-testing, _if_ the goal is actually to beta-test, and not just to get some free publicity: I'd rather have just 50 people actually professionally looking for bugs, than 50,000 whining about everything else"

        Well, sure if you have that exact goal...but that's not what beta testing is.

        Lets say that your scope was to take those 100,000,000,000 pages and check them for errors and typos...now which tea

        • I'm not knocking the idea of the seasoned beta tester...they're important as well. They know what to do, and they do it more efficiently and more systematically, but they can't do *everything*. They can't stress the systems as much as a crowd of fanboys can.

          That is what I am saying. Let the beta testers that do it for a living get the job done and get the software solid. Then hit the stress test to get as many possible combinations of PCs running the software and hitting the servers to get those bugs out
  • by AuMatar ( 183847 ) on Tuesday October 18, 2005 @06:44PM (#13821988)
    In that order. You don't get many useful bug reports from the large betas (you do from the smaller stages), but you don't know how it will handle release type number of users until release, unless you beta it.
  • by incubusnb ( 621572 ) on Tuesday October 18, 2005 @06:46PM (#13822006) Homepage Journal
    perhaps you don't understand the term "Load Testing", to properly stabilise a persistant world you have to put the servers under conditions that are similar to how they would be after release. The only real way to Load test an MMO is to have actual people playing the game, sure, you could populate the world with a basic AI, but you wouldn't get the same situations that a human would get into. perhaps the Article Writer knows next to nothing about MMO development and is just pissed off because he couldn't get a Beta copy of City of Villians or something.
    • I understand load testing... hence why I pointed out that is what the MMORPG developers need to do. Skip all these baby step "only a few in at a time" elitist testing phases. Drop 1,000 players in a day and keep going until the servers don't work anymore. The whole process of selecting a few select beta testers only wastes time and resources which could be better placed into the back end bug reporting systems or fixing the actual bugs themselves. Darniaq pointed out something on his blog [darniaq.com] that I totally
      • Hmmm. Well in the two betas I'm in at the current moment it kind of went this way:

        Initial limited beta invites were a few thousand. This initial load revealed some big problems with the login system and database on both games. At this point, the beta sessions where more like "crash the server super quick to generate a ton of debug info". It seemed like in both cases the initial limited beta helped work out some major scale issues with the primary logon and character databases. This round of beta was also us
        • Exactly why I believe that early beta or alphas are best tested by profesionals that are paid to do so.

          Then you drop 1000 players in at a time and nail out the bugs such that you mentioned in your beta experience.
          • I'm a professional programmer with 20 years experience, and am good at producing detailed steps at reproducing problems. I have stated so on numerous alpha and beta test applications. I have had old machines at the dividing line of the recommendation, and the very latest Alienware with top end 3D card.

            I have never been accepted, except for "taking everyone who applies" open betas.

            So I really doubt they necessarily screen for quality testers. Or believe the applications. Or care.
  • Stress Test (Score:3, Insightful)

    by king-manic ( 409855 ) on Tuesday October 18, 2005 @07:10PM (#13822243)
    There are a few things you will find difficult to stress test properly. It takes a full scale assault by potential users to see how well your hardware infrastucture stands up. In single player games this is a non-issue. In even multi server small scale multiplayer games it's a non issue as well. But when you have 64+ connections to a server then you have to see if you theoretical test bear out in reality.
  • by wyldeone ( 785673 ) on Tuesday October 18, 2005 @07:16PM (#13822292) Homepage Journal
    The point of betas of mmorpgs is advertising. Nothing more, nothing less. It is very difficult to actually test the games, let alone have your suggestions heard in the environment set up by the game companies. They serve the same purpose as game demos released a few weeks before the release of a prominent single player game, which is to drive excitement and anticipation of the final product. I am part of a beta testing group for Activision, which stays together from game to game, and is a smaller, more intimate group. We are able to actually test and improve games (we have worked on COD, COD:UO, RTW, THUG 2, and many others published by the company), but in the environment produced by mmorpg companies this is not the goal.
    • Exactly. Betas for MMOs have become just another marketing gimick. Need proof? Just look at the release of City of Villains. Websites are using beta passes to drive subscribers, and the company is using it to drive preorders.

    • I've noticed that single player demos have dropped off in the last couple of years. They are either not released or released quite a while after the game is. I can't imagine that it's that difficult to released a demo and it's free publicity. Personally it's very rare that I'll buy a game without playing it first.
      • I've learned this the hard way. Ironically, I went and bought Black & White (and expansion pack!) based on it being one of the most highly rated games of all time. And I just couldn't get into it. I wasn't into "training" my giant monster (the sheep!). And the RTS aspect of it was somewhat lacking as I hated having to use the sheep to do a lot of stuff. I prefer the units highly automated so I don't have to micromanage stuff.

        So, too, with Warcraft III. Just didn't do it for me. It seemed like a R
  • Wow showed that load testing is pretty damn important. I douby a single user can really simulate all the factors that go into something like that.
    • True, however solo play does also need to be tested. Players need to have an option when their friends are on. Pick-up groups are the single biggest cause of account cancellations right after grief players (and sometimes it's because of grouping WITH grief players). The games that I'm betaing now (can't say due to NDA) I'm concentrating on testing solo play, to make sure that customers that prefer that playstyle "most" of the time have an option. Obviously, some things in these games will require groups. Ba
      • should have been "are not on" thanks slashdot for the delay after posting. Even after first review I didn't catch that until I hit the post button...
      • Pick-up groups are the single biggest cause of account cancellations
        This is why I stopped playing WoW for almost 6 months. Horrific PUG experiences with my 50+ Hunter. (I wasn't guilded) Unfortunately, there is nothing the Devs can do to make players not suck to play with. The only reason I'm back in game is that myself and 4 friends all started new characters at the same time, and we always play together. I'll never have to do a PUG again!
  • One thing that, while rare, is still quite aggravating is when a studio alpha tests with next to no features and then charges for the beta. I still can't find out how this does anything but alienate users, but somehow people manage to pull it off.
    • by Bad Ad ( 729117 )
      except that they dont charge for betas, as they wipe all your account after the betas ended.
  • by Tojosan ( 641739 ) on Tuesday October 18, 2005 @09:24PM (#13823220)
    It's like giving out free maryjo samples.....a dealer is bound to get some instant new business. Heck, they could get a commercial on every TV show for free and it wouldn't be near the value of 1000 potential customers getting a taste. And not a taste with 1 million others but a very tight availability taste.

    Also, as a programmer, I can say you can unit test till you are blue in the face, but it only takes a user 5 minutes to find a bug you'd have never tested for.
    If only one major bug is caught that's huge icing on the marketting and catching customers angles.

    But I think there is one factor we overlooked...ego. If you'd spent years developing a new toy, you didn't do it to keep it locked up for yourself. You did it for someone to play with. I'm betting the rush of first players digging into the new toys you rolled out is huge! and heck, if it sucks, at least you only had to deal with the a small amount of laughs the first day. Ha!

    Be swell
  • Response (Score:2, Insightful)

    by Darniaq ( 738858 )
    I WISH beta processes were more harrowing to get into. Without a more stringent front end, we'll just perpetuate this cycle of useless noise drowning out any real hope of relevant reporting.

    From here [darniaq.com]:

    There needs to be a good back end reporting system too. Forums do not cut it. They may work for a few hundred testers, of which maybe 75% of them would read the forums and 50% actually post. However, when the game starts stress testing the servers, the players will generate much more noise than actual sig
    • I linked your post up further there Darniaq and had tried to edit the /. submission to include it since it brings up a great point.

      However I am still a firm believer that if you're going to be that strict on picking testers you can put that money and man hours into paying real game testers.

      The public community of gamers is best at stress testing... at which point a focused and coordinated bug reporting system is required.
    • "In my opinion, all reporting should be done through ingame interface. And, the reporting should be based on the developers pushing specific agendas for specific results, at least most of the time."

      This is how I've seen it done in a couple of betas I've been on. The bug submittal is done through a game command or interface option so you can log the bug as it happens. Additionally, important information such as the player's location, status and even video card used can be automatically gathered.
      While the tes
  • The problem with the article is that the author is assuming that the beta tests are something that they are not intended to be. The idea that the testing and late stage development of a multiplayer product is analogous to that a single player game shows a gross misunderstanding of the difference in the development process of multiplayer games.

    MMO beta tests are not for fielding player responses, taking suggestions from the public, or even for bug reporting. The development team and internal QA does all of t
  • by MMaestro ( 585010 ) on Wednesday October 19, 2005 @12:31AM (#13824101)
    So now every new beta application is going to be over stating what hours they play or the person submitting it will try to *guess* the *magic combination* of inputs to produce the highest % chance of getting into beta. Congratulations Sigil; you just flooded your beta application pool with a bunch of false information.

    Actually some MMO players generally UNDERESTIMATE what hours they play, when and how powerful/weak their PC rigs are. You have to consider the users who stay logged on 24/7 'because they can' or because they run AFK player-run shops or whatever. If the servers stay up 24/7, theres someone out there who will stay online 24/7.

    How much work is it to review countless beta applications? I have no solid numbers, but there is no way they can convince me that it doesn't take away from the game development.

    Plug data into an Excel worksheet, randomly pick X number out and thats Group 1. Sort the data by 'average time spent playing games' and thats Group 2. Etc, etc, etc. Any programmer who can write a MMO game can write a program that can automate this.

    The idea of NDAs is also hard for me to understand. World of Warcraft had no problem without one.

    Actually, WoW didn't have a NDA because that wasn't a beta. It was free marketing. And the 'closed beta' prior to it DID have a NDA. Given how poorly the servers did at launch day, the fact that they SERIOUSLY did not take that last 'beta' seriously is clear.

    WoW beta only suffered from too much interest, but Blizzard did a remarkable job of eventually getting 500,000 testers online.

    Bolding by me.

    Sigil will be balancing this game as any other MMORPG... over time!

    Yeah other MMO games like WoW and FFXI tried before and nearly killed themselves. WoW's player run economy is a joke since anyone who can use a keyboard could craft and all but the most extreme basic materials were sold by NPCs. On top of that people were hitting the level 50 limit within weeks of the games launch and the best equipment is all obtained from time consuming, pro-hardcore instances with drop rates on par with winning the lottery. Its the exact opposite with FFXI. Leveling up in FFXI is considered to be the worst 'grind' in out of every other MMO out there, crafting is a near impossibility without learning economics 101 and accounting 101 not to mention the Chinese 'gilfarmers' screwing around with inflation. The best equipment either costs more money than your day job's wages or is dropped from an impossible to solo monster and probably has a bad drop rate.

    The general rule of thumb for MMOs is that time does not cure all. Give WoW a year and people will be bitching about lack of things to do after hitting level 60 on every class and race combination. Give EQ2 a badly designed economy and one year later SOE will start acting desperately... oh oops, that already happened.

    • No I just blog about MMOs because I don't play them o_O

      Anyways. Actually some MMO players generally UNDERESTIMATE what hours they play, when and how powerful/weak their PC rigs are. You have to consider the users who stay logged on 24/7 'because they can' or because they run AFK player-run shops or whatever. If the servers stay up 24/7, theres someone out there who will stay online 24/7.

      -24/7 loggers are not what make a beta test run. Its the person that plays the game for an hour and analyzes his pl
  • Sign up on www.betaguild.com [betaguild.com] . I know it hasn't been updated for a while, but thats because my host is undergoing renovation and I can't change stuff. I can get you a part of the team though. I'm looking for a crack team of beta testers. In the long run, I'll figure out who's the best beta testers around, and be able to solicit them to companies. For right now, we're just in the slow growth stage for a few years.
  • The author acts as if this phase of beta is a stress test. This is very early beta. It's basically the Alpha stage of WoW. The two game companies just choose to use different terms for the phases of beta. There are many game mechanics that are going to change in this stage of beta. For example one thing Brad wants to test is Trivial Loot Code(TLC). When so much is changing and the server structure is still under development you want to be real selective on who is playing the game and giving you feedback.
    • To quote one of my favorite movies ever... "You are not your list of games tested." You are reding the /. quote and not the article. My view is that selective beta processes waste time. There is very good software beta testing companies out there that are reasonably priced, specialized in breaking software, and provide better feedback than 99% of any *over the net* tester you will get. This is a process that has worked for countless years in the single player market. It is ignorant to say MMORPG != sin
      • I did read your entire article. Why don't you read my entire comment. You want them to go right to stress testing. The game is no where near ready for that. The game mechanics and many other things still need to be worked out. From your attitude it sounds like you have never even participated in a small exclusive beta test. Many important things are done in this phase and it is a neccesary step for these types of games.
        • Actually you missed my point then. I am for replacing the early *small exclusive beta test* with a profesional beta testing company or in house testing groups(for larger companies). I agree 100% many important things are done in that phase.

          The issue I have is that the amount of time and resources wasted trying to pick a few select beta testers out of a pool of thousands is better spent developing other areas... such as what Darniaq has suggested in the actual bug reporting functions.
  • Recently, Turbine announced 125 000 (!!!) applications for Dungeons & Dragons Online alpha. Now that's ridiculous. It's a marketing trick to make everyone believe that, even in early stages of development, the game seems interresting to a lot of people.

    Then you have Blizzard, who didn't requested a NDA for World of Warcraft. This way, a lot of people would make movies, screenshots, features reviews, etc... Another marketing trick to create a lot of hype.

    And you have Sigil, who allows only the people

  • ...I have participated in many betas(and quite a few alphas). The one things that I have learned is that the developers never seem to listen to what the players tell them. One of the games myself and many other testers pointed out a HUGE flaw in the game balance, 4 months before it went live. It was not until several months after go live that they eventually fixed it.

    The developers should be more willing to listen to testers feedback about the games. In my opinion testing is not just about finding bugs
    • Yeah, that's what I was suspecting/wondering. Some games have been launched with such glaring errors, that, yes, I've always wondered if anyone actually tried finding any bugs during their much-hyped beta.

      Heck, while you have a point that UO _is_ a good example there, I have an even better one: Anarchy Online. Read the review on Something Awful, and I can personally atest that yes, the game was _that_ broken after the devs claimed it was finally 110% fixed and stable. In fact, there's a whole slew of of mor
    • Developers will not receive quality feedback from players until they start giving some of their own. When a player submits a bug report, no matter how detailed, how obvious, how easily-reproduced, how easily-corrected it is -- what happens? Nothing. No questions back to the player, no fix produced, not even acknowledgement of receipt. I honestly don't see the point of beta programs or official bug reporting mechanisms when they are routinely ignored by the developers.

      A person like me would say the industry
  • but why even have such a process in the first place? If they honestly think they are going to get any sort of actual *testing* (I use the term loosely) from an over-hyped MMORPG community... they obviously failed basic MMORPG sociolog

    Heh, but blizzard changed a lot pre-launch to post launch of wow. For those that play, the paladin class was totally revamped (many believe for the worse), and a lot of balancing issues were worked out (Will of the forsaken was originally always on, Mortal strike did 200%
    • Attributed to Blizzard's class balance internal testing team in their very own QA office. That is how Blizzard determines and reviews each of the WoW classes... internally. Player feedback appreciated... left in the o-file.

      Testing... as in finding bugs that stop the game from being playable. Balancing is something that will happen pre-release, post release, and further down the road regardless of beta testing. The bugs will be fixed hopefully, but the balance will be lacking.

      Not saying these areas
  • New Lows (Score:3, Informative)

    by thebdj ( 768618 ) on Wednesday October 19, 2005 @10:34AM (#13826416) Journal
    Wow, now we are having bloggers post links to their own blogs to drive up traffic...and to make it worst the thing is hosted on blogspot. I mean seriously, at least get your own DNS and route it or buy some shared spaced.

    Now onto the topic at hand, this just shows that 90% of bloggers are talking out of their ass. In this case he really misses the point of beta testing in an MMO. There are several very good reasons and they are reasons that are necessary to test. Later stages in almost all MMO beta test are load tests. They could care less at that point who reports bugs, they just want to make sure the servers don't go kaput when the game launches.

    Another problem is you need a large number of people to beta test any MMO. There are literally hundreds of possibilities when a game is in the beta mode (if not thousands) for character development. Look at World of Warcraft. You have to test every race with every classes. You have to test quests at multiple levels, you need to test raids and dungeons. There is a lot more to beta in a MMO then in a single player or even regular multi-player game.

    Not to mention the fact that the Beta taste gives you the chance to hook all those players and then start charging them. It also gives you the chance to get free word of mouth advertising. Open betas are a bit more of a joke on other games, but in those situations companies release "demos" to perform essentially the same task without calling it a beta. I have beta tested a regular game and an MMO. Let me say it is much bet to use a community of people in the tens or hundreds of thousands to test your game then to use your few hundred employees with an MMO. Like I said before, the sheer scale of MMOs makes fully closed Beta's an unreliable and ineffective means to test the game.
    • Congrats on actually reading the article and not just the /. blurb.... oh wait you didn't. Read my various other replies for the answers you seek.

      Every single thing you stated is stated in my article. Congrats on repeating it! Also read the previous posts on how quantity of testers does not equal quality.

      Now say it with me... "Before I post on /. about an article, I will actually read it." Now don't we all feel better?
    • Also if you would like I will have a *real* website host it... then would my credibility score rise with you? Seriously... stop looking at the URL, read the actual aticle, and evaluate it based on how it is written.
  • I can't tell what the point of this article is. It looks like the author was pissed off by how long the Vanguard beta sign up form was and decided to rant about it. The few points he makes are wrong, anyway.

    Sigh... now all of the good articles about VG beta won't get airtime. For shame.

    I hereby dub the author of that article Mini_Jack_Thomson_09572 , for his ability to say nothing useful yet still get media attention.

  • Most of the players treat it as a sneak preview at best or more commonly just free play.

    A tiny fraction of people find and report bugs as they're supposed to.

    A much larger minority actively looks for glitches, bugs, and exploits ... and then proceeds to squirrel them away for later. Then at go-live time the carefully-planned game balance is instantly pooched by these guys who run straight to the loophole and max out while the new players are still trying to find the bank.

    I can see a marketing reason to ope
  • What you need to know is the fact that betas are infiltrated by those that want sneak peaks at the game. Definitely not by those that truly wish to test the product.

    They're hardly being "infiltrated" by anybody. If you pre-order most games these days, you get guaranteed beta access which oddly coincides with when they're ramping up their stress tests. If you already subscribe to an online game the company is running, you tend to get beta access to their new stuff as well (whether or not you fork out cas

I have hardly ever known a mathematician who was capable of reasoning. -- Plato

Working...