Overcomplicated MMO Betas 64
Heartless writes "On the heels of Vanguard's beta 1 announcement, Heartless Gamer blog has an article looking into why MMO beta processes are overly involved and detracting from the game they are meant to improve. From the article: 'But why even have such a process in the first place? If they honestly think they are going to get any sort of actual *testing* (I use the term loosely) from an over-hyped MMORPG community... they obviously failed basic MMORPG sociology. I could link hundreds of beta leaks and broken NDA contracts, but what would be the point? What you need to know is the fact that betas are infiltrated by those that want sneak peaks at the game. Definitely not by those that truly wish to test the product. Internal testers and paid testers have proved for years to be able to produce very finished products in the single player market.'"
Gimmicks (Score:3, Insightful)
Re:Gimmicks (Score:1)
Sufficed to say, we all
Re:Gimmicks (Score:2, Informative)
This has been the bane of many MMO's that have come and gone. Pushed to get the game out too soon, rushed into a buggy beta test that reveals nothing but how premature the
game really is, it ruined Earth and Beyond for example, beta'd too early, too buggy, too unbalanced and for too damn long (linke 6 months or more of open beta,
Weight Of Numbers (Score:4, Insightful)
1% genuinely beta test
That's a 100 person QA team - far bigger than the typical MMO will ever see.
Now up those numbers to:
50,000 players...
1% genuinely beta test
5-10% vocally bitch about every weird bug and quirk they find
Now you're looking at 500 decent QA testers and another 2,500-5,000 pain in the ass guys who're maybe worth 1% of a tester each but cumulatively do still add up.
A beta test doesn't have to have every player responsibly beta testing. Sheer numbers ensure the end effect still gets met.
Besides, by public beta, the main thing that should be getting tested is load and the weird load quirks caused by 5,000 players all deciding to try the same exploit etc. That, whether they're good testers or bad, still happens. Arguably it happens even better if they're "bad" beta testers as they're more likely to do things they "shouldn't".
Re:Weight Of Numbers (Score:4, Interesting)
Re:Weight Of Numbers (Score:2)
I know long reports are hard on the QA folks, but for a real issue it can be necessary.
A lot of players submit bug reports on stuff they just don't understand too. It's part of the game, but they havn't realized it yet (my fireballs don't work on this door!).
Noise To Signal Ratio (Score:4, Insightful)
In other words, when I Google for something, I'd rather have 1 link that is exactly what I want, than 100,000,000,000 irrelevant links. The same goes for beta-testing, _if_ the goal is actually to beta-test, and not just to get some free publicity: I'd rather have just 50 people actually professionally looking for bugs, than 50,000 whining about everything else.
Having 500 people who genuinely test for bugs, is _worthless_ if their signal is drowned in the noise from 50,000 people posting like there's no tomorrow about how your game sucks ass because his Priest doest't _start_ with the Mages' level 50 spell. (That's sadly not even a joke. Something Awful once had a parody of an open letter to Sony, in which they asked for really ludicrious stuff, including _literally_ that a level 1 priest should start with the most powerful mage spell. Much to their surprise, they got a helluva bunch of emails aggreeing wholeheartedly.) Or how it sucks ass and is unbalanced because it doesn't _force_ everyone else to group with his Priest that bought everything _except_ healing/buff spells. (Add a long circular-backpatting whine about how players are idiots and don't appreciate how useful that priest is with his mace alone.) Etc.
And it goes downhill from there. The guy who discovered a bug and filed it, will start _one_ post. The guys arguing that their characters should have 100% resistance to damage and an insta-kill spell that costs no mana, will start one per day. And more often than not, spill into the other topics too. (Surely a post about how a mage spell sometimes fails with no explanation, not even a "your spell was interrupted" message, is _the_ right place to post about how either (A) you mages had it too good and it was about damn time that spell got a downgrade, or (B) about how we mages are the whipping boys of the devs, and they downgraded yet another of our spells. Doom, gloom, run for the hills, and all that.)
Welcome to the wonderful world of looking for the proverbial needle in the haystack.
Re:Noise To Signal Ratio (Score:3, Insightful)
Well, sure if you have that exact goal...but that's not what beta testing is.
Lets say that your scope was to take those 100,000,000,000 pages and check them for errors and typos...now which tea
Re:Noise To Signal Ratio (Score:1)
That is what I am saying. Let the beta testers that do it for a living get the job done and get the software solid. Then hit the stress test to get as many possible combinations of PCs running the software and hitting the servers to get those bugs out
Marketing and stress test (Score:4, Interesting)
Public Beta testers aren't bug-hunters (Score:4, Insightful)
Re:Public Beta testers aren't bug-hunters (Score:2, Interesting)
Re:Public Beta testers aren't bug-hunters (Score:2)
Initial limited beta invites were a few thousand. This initial load revealed some big problems with the login system and database on both games. At this point, the beta sessions where more like "crash the server super quick to generate a ton of debug info". It seemed like in both cases the initial limited beta helped work out some major scale issues with the primary logon and character databases. This round of beta was also us
Re:Public Beta testers aren't bug-hunters (Score:1)
Then you drop 1000 players in at a time and nail out the bugs such that you mentioned in your beta experience.
Re:Public Beta testers aren't bug-hunters (Score:1)
I have never been accepted, except for "taking everyone who applies" open betas.
So I really doubt they necessarily screen for quality testers. Or believe the applications. Or care.
Stress Test (Score:3, Insightful)
That's not the point (Score:5, Insightful)
Re:That's not the point (Score:2)
Re:That's not the point (Score:1)
Re:That's not the point (Score:1)
So, too, with Warcraft III. Just didn't do it for me. It seemed like a R
single player != mmorpg (Score:1)
Re:single player != mmorpg (Score:2)
Re:single player != mmorpg (Score:2)
Re:single player != mmorpg (Score:1)
*quits slashdot because he can't find a single decent group of people to converse with and is bothered by all the griefers*
PROOF'd
Re:single player != mmorpg (Score:2)
Re:single player != mmorpg (Score:2)
Besides, we've only just hit 30. We can recruit 5 more likeminded individuals as we get closer to that point.
Re:single player != mmorpg (Score:2)
In direct contrast, a guild we play with sometimes has recently started to blindly recruit. They are going for the numbers in order to run Molten Core and Zul Gurrab. Some of their new players are complete idiots and on more than one occ
Yet more gripes from the land of betas (Score:2)
uh? (Score:1)
More than free advertising... (Score:4, Interesting)
Also, as a programmer, I can say you can unit test till you are blue in the face, but it only takes a user 5 minutes to find a bug you'd have never tested for.
If only one major bug is caught that's huge icing on the marketting and catching customers angles.
But I think there is one factor we overlooked...ego. If you'd spent years developing a new toy, you didn't do it to keep it locked up for yourself. You did it for someone to play with. I'm betting the rush of first players digging into the new toys you rolled out is huge! and heck, if it sucks, at least you only had to deal with the a small amount of laughs the first day. Ha!
Be swell
Response (Score:2, Insightful)
From here [darniaq.com]:
There needs to be a good back end reporting system too. Forums do not cut it. They may work for a few hundred testers, of which maybe 75% of them would read the forums and 50% actually post. However, when the game starts stress testing the servers, the players will generate much more noise than actual sig
Re:Response (Score:1)
However I am still a firm believer that if you're going to be that strict on picking testers you can put that money and man hours into paying real game testers.
The public community of gamers is best at stress testing... at which point a focused and coordinated bug reporting system is required.
Re:Response (Score:2)
This is how I've seen it done in a couple of betas I've been on. The bug submittal is done through a game command or interface option so you can log the bug as it happens. Additionally, important information such as the player's location, status and even video card used can be automatically gathered.
While the tes
The value of public testing... (Score:2, Insightful)
MMO beta tests are not for fielding player responses, taking suggestions from the public, or even for bug reporting. The development team and internal QA does all of t
Has this guy played a MMO? (Score:3, Interesting)
Actually some MMO players generally UNDERESTIMATE what hours they play, when and how powerful/weak their PC rigs are. You have to consider the users who stay logged on 24/7 'because they can' or because they run AFK player-run shops or whatever. If the servers stay up 24/7, theres someone out there who will stay online 24/7.
How much work is it to review countless beta applications? I have no solid numbers, but there is no way they can convince me that it doesn't take away from the game development.
Plug data into an Excel worksheet, randomly pick X number out and thats Group 1. Sort the data by 'average time spent playing games' and thats Group 2. Etc, etc, etc. Any programmer who can write a MMO game can write a program that can automate this.
The idea of NDAs is also hard for me to understand. World of Warcraft had no problem without one.
Actually, WoW didn't have a NDA because that wasn't a beta. It was free marketing. And the 'closed beta' prior to it DID have a NDA. Given how poorly the servers did at launch day, the fact that they SERIOUSLY did not take that last 'beta' seriously is clear.
WoW beta only suffered from too much interest, but Blizzard did a remarkable job of eventually getting 500,000 testers online.
Bolding by me.
Sigil will be balancing this game as any other MMORPG... over time!
Yeah other MMO games like WoW and FFXI tried before and nearly killed themselves. WoW's player run economy is a joke since anyone who can use a keyboard could craft and all but the most extreme basic materials were sold by NPCs. On top of that people were hitting the level 50 limit within weeks of the games launch and the best equipment is all obtained from time consuming, pro-hardcore instances with drop rates on par with winning the lottery. Its the exact opposite with FFXI. Leveling up in FFXI is considered to be the worst 'grind' in out of every other MMO out there, crafting is a near impossibility without learning economics 101 and accounting 101 not to mention the Chinese 'gilfarmers' screwing around with inflation. The best equipment either costs more money than your day job's wages or is dropped from an impossible to solo monster and probably has a bad drop rate.
The general rule of thumb for MMOs is that time does not cure all. Give WoW a year and people will be bitching about lack of things to do after hitting level 60 on every class and race combination. Give EQ2 a badly designed economy and one year later SOE will start acting desperately... oh oops, that already happened.
Re:Has this guy played a MMO? (Score:1)
Anyways. Actually some MMO players generally UNDERESTIMATE what hours they play, when and how powerful/weak their PC rigs are. You have to consider the users who stay logged on 24/7 'because they can' or because they run AFK player-run shops or whatever. If the servers stay up 24/7, theres someone out there who will stay online 24/7.
-24/7 loggers are not what make a beta test run. Its the person that plays the game for an hour and analyzes his pl
If you want to get paid to beta test (Score:2)
Beta Tests (Score:1)
Re:Beta Tests (Score:1)
Re:Beta Tests (Score:1)
Re:Beta Tests (Score:1)
The issue I have is that the amount of time and resources wasted trying to pick a few select beta testers out of a pool of thousands is better spent developing other areas... such as what Darniaq has suggested in the actual bug reporting functions.
Sigil is not like Blizzard or Turbine (Score:2)
Then you have Blizzard, who didn't requested a NDA for World of Warcraft. This way, a lot of people would make movies, screenshots, features reviews, etc... Another marketing trick to create a lot of hype.
And you have Sigil, who allows only the people
They don't listen anyway.... (Score:2)
The developers should be more willing to listen to testers feedback about the games. In my opinion testing is not just about finding bugs
That's another thing I was wondering (Score:2)
Heck, while you have a point that UO _is_ a good example there, I have an even better one: Anarchy Online. Read the review on Something Awful, and I can personally atest that yes, the game was _that_ broken after the devs claimed it was finally 110% fixed and stable. In fact, there's a whole slew of of mor
Re:They don't listen anyway.... (Score:1)
A person like me would say the industry
Heh (Score:2)
Heh, but blizzard changed a lot pre-launch to post launch of wow. For those that play, the paladin class was totally revamped (many believe for the worse), and a lot of balancing issues were worked out (Will of the forsaken was originally always on, Mortal strike did 200%
Re:Heh (Score:1)
Testing... as in finding bugs that stop the game from being playable. Balancing is something that will happen pre-release, post release, and further down the road regardless of beta testing. The bugs will be fixed hopefully, but the balance will be lacking.
Not saying these areas
New Lows (Score:3, Informative)
Now onto the topic at hand, this just shows that 90% of bloggers are talking out of their ass. In this case he really misses the point of beta testing in an MMO. There are several very good reasons and they are reasons that are necessary to test. Later stages in almost all MMO beta test are load tests. They could care less at that point who reports bugs, they just want to make sure the servers don't go kaput when the game launches.
Another problem is you need a large number of people to beta test any MMO. There are literally hundreds of possibilities when a game is in the beta mode (if not thousands) for character development. Look at World of Warcraft. You have to test every race with every classes. You have to test quests at multiple levels, you need to test raids and dungeons. There is a lot more to beta in a MMO then in a single player or even regular multi-player game.
Not to mention the fact that the Beta taste gives you the chance to hook all those players and then start charging them. It also gives you the chance to get free word of mouth advertising. Open betas are a bit more of a joke on other games, but in those situations companies release "demos" to perform essentially the same task without calling it a beta. I have beta tested a regular game and an MMO. Let me say it is much bet to use a community of people in the tens or hundreds of thousands to test your game then to use your few hundred employees with an MMO. Like I said before, the sheer scale of MMOs makes fully closed Beta's an unreliable and ineffective means to test the game.
Re:New Lows (Score:1)
Every single thing you stated is stated in my article. Congrats on repeating it! Also read the previous posts on how quantity of testers does not equal quality.
Now say it with me... "Before I post on
Re:New Lows (Score:1)
I am of the concerned variety that believe resources and time are being wasted on overcomplicating the beta process. That same time and resources can be put back into improving the back end bug
Re:New Lows (Score:1)
Re:A Different concept (Score:1)
Someone didn't get in (Score:1)
Sigh... now all of the good articles about VG beta won't get airtime. For shame.
I hereby dub the author of that article Mini_Jack_Thomson_09572 , for his ability to say nothing useful yet still get media attention.
What actually happens in beta (Score:1)
A tiny fraction of people find and report bugs as they're supposed to.
A much larger minority actively looks for glitches, bugs, and exploits
I can see a marketing reason to ope
Wha-huh? (Score:2)
They're hardly being "infiltrated" by anybody. If you pre-order most games these days, you get guaranteed beta access which oddly coincides with when they're ramping up their stress tests. If you already subscribe to an online game the company is running, you tend to get beta access to their new stuff as well (whether or not you fork out cas