Exploring the Current State of Beta Testing 82
Karen Hertzberg writes "Since the earliest days of MMO gaming, beta testing has played a pivotal role in the success or failure of our persistent worlds. We've come a long way since the initial tests of Ultima Online and The Realm, but what role do our current beta tests play in the potential outcomes of unreleased titles? To answer this question, Ten Ton Hammer turned to current and former beta decision makers at Cryptic Studios, NetDevil, Sony Online Entertainment, Funcom, and Mythic Entertainment. Some of their answers — and the information they reveal — may surprise you."
Re:There Really Are Some Gems in This Article (Score:5, Interesting)
Opposite reaction in Guild Wars (Score:5, Interesting)
I was part of the open beta of guild wars(i.e. i pre-ordered and was part of the PR wave of beta). It was awesome, everything was fun, it was clear that it wasn't finished, but the missions were OK, the PvE was tolerable, but the PVP was phenomenal. When it was time to release, I fired up my copy and found like 2 skills at the first skill trainer. Approximately 750 in game hours later, it was possible to recreate the PVP experience I had during beta...
750 hours in missions that are only OK and tolerable PvE that turned to miserable at the snails pace that they made you try it. Guildwars isn't a monthly thing based fee, so they gained nothing, absolutely zero, by forcing you to put 750 hours into the original campaign to get back to the fun of the open betas. By then, they had lost a very large portion of their user base and the beta users were not the the majority of the major adopters. If they had released the game we "beta tested", it probably would have been a runaway success instead of the third rate game it is today. Also, because the PVP players left for greener pastures (battlefield 2 so you can have an idea of what that crowd was), current PVP metagame is a pale imitation of what it once was and could be.
For the record, about 2 years after release, they made enough changes so that a new player can jump into that old timey fun.
What's really funny? Cryptic's comments. (Score:1, Interesting)
Their comments in the article are completely at odds with my experience with CO, and the experience of several other former closed beta testers I've spoken with.
Their early closed beta was ugly. UGLY (Bolded, and blown up to 70 foot high glowing neon letters).
Stuff was just nonfunctional, or intermittently functional. The game was unstable, even on hardware of exactly (or above) recommended spec. They had a whopping one zone in a semi-finished state.
Bugs were ignored through repeated revisions. Stability issues were met with intimations that you shouldn't be on the beta and completely non-veiled threats to boot you off if you didn't just shut up, play, and give glowing feedback.
I played through several months of release before I finally gave it the middle finger.
It could be worlds better now. I don't care. I won't give them a dime.
Re:Halfway just for PR, Bad "beta testers" (Score:4, Interesting)
I think if game companies really wanted to get their value out of beta testers, they would implement simple, trivial changes like requiring a questionnaire after every N hours of game time. I'm not saying bug reports, but at least a 10-minute form where developers can ping the gamers for feedback, at least once or twice weekly.
If the kids can't take 10 minutes of their time to help improve the game they sink 20 hours a week into, they can fuck right off and wait for retail. They will buy it anyway, and if they don't, well fuck 'em. That's business.
Re:There Really Are Some Gems in This Article (Score:5, Interesting)
I don't know, I've been in closed beta for several products lately, Warhammer Online, Age of Conan and Tabula Rasa.
I submit bug reports out the wazoo (and believe me, all 3 of these have had a few bugs). I never see anything come of them, even the major ones. I try to view being in the beta as a privilege for which I owe good bug reports.
On the other hand, you are supposed to have fun. Any time I am not having fun, is just as much of a bug as when things don't work.
I've always got the feeling that the game was going to ship no matter what I found. I was just there to load the servers and stress the network so they could make the launch less hokey. I've found blatant exploits, item dupes, countless serious quest bugs, etc. I never saw any movement on any of those, nor was asked to clarify or repro. That kind of tells me a lot about how my input is being considered. It really is just a marketing thing that the developers have been able to put to small use. Beta's rarely start early enough, nor is there sufficient in-house support for beta testers for it to really be effective.
Does anyone remember what "beta" meant? (Score:3, Interesting)
There was a time when alpha meant "feature-complete, but broken." Beta meant "OK, let's get the last few bugs out before release."
Later, as code got more complex, beta usually went through a few phases. That was fine. Also, the beta testers were generally professionals, with some exceptions. (it's always a good plan to get real users who can break the unbreakable. Just make sure you don't count on them exclusively.)
Nowadays, what companies put through 'beta testing' is rarely alpha-code. Feature complete? Maybe, maybe not. Realistically, the second release candidate onwards through the first post-release patch should properly be considered beta, because the number of products that are even usable until after the first patch are minimal.
Ultimately, quality code doesn't pay in almost all commercial cases. Get enough untrained end-users to find the worst of the show-stopper bugs, and then release the code and start making money. Once in a long while, your product will be so bad that it falls on its face--but that's the exception, and will probably still cost less than professionally beta-testing all of your products to a high level of quality.