Exploring the Current State of Beta Testing

Karen Hertzberg writes "Since the earliest days of MMO gaming, beta testing has played a pivotal role in the success or failure of our persistent worlds. We've come a long way since the initial tests of Ultima Online and The Realm, but what role do our current beta tests play in the potential outcomes of unreleased titles? To answer this question, Ten Ton Hammer turned to current and former beta decision makers at Cryptic Studios, NetDevil, Sony Online Entertainment, Funcom, and Mythic Entertainment. Some of their answers — and the information they reveal — may surprise you."

Re:There Really Are Some Gems in This Article

[Chabo]

Beta tests now are glorified demos for the games. It used to be so much different. I really hate what's been done to them, and the only way to be a 'real' tester now is to get into the alpha's or in some rare cases early closed betas.

Backing up the first quote and pointing out what Betas have become.

IMO, the point of an open beta isn't to make client-side changes, because the developers can make all of those changes based on alpha or closed-beta testing, much earlier in the development process. By the time a product reaches beta stages, it's essentially done, and just needs a small amount of polish before release.

The real point of an open beta, especially for MMOs, is stress-testing the game servers. An open beta by definition tries to get everyone possible to play the game, so in addition to being a demo for the game, you're also trying to debug any bugs in the server system that simply can't be found by testing with just a few clients.

This was about all Blizzard was concentrating on with WoW during the open beta, and we can even see this outside the genre; when Valve had the pre-release demo of Left 4 Dead, they were testing what happens when thousands of gamers used their brand new matchmaking system. It had some issues that have been largely resolved by now, but they just couldn't test it that easily without that kind of hammering.

Once Beta, it's Done

[MarioMax]

At least in the case of SOE, once a game is in Beta, it's basically done. They just want marketing and bug reporting. The devs are extremely reluctant to change anything once their product is in Beta, no matter how fundamentally flawed the product is in the first place.

Halfway just for PR, Bad "beta testers"

[TibbonZero]

By the time most of these games hit Public Beta it's really just stress testing the servers a bit (but never enough) and working on some tuning stuff. Mainly I've seen this by this stage it's really just about getting a solid buzz going around the game. Most users will have broken their NDAs by this point (as we saw happen recently in WotLK), but the entire point is just to get hype going for the game.

Until recently I was working for a game-industry related company, and we had a lot of close interaction with gamers and the game companies. I'm reading the article fully right now for some more of the developer/publisher-perspective details however.

Half the problem is that most of these gamers suck at betatesting. They don't want to file bug reports, they want to play the game free/early so that their guild can get a head start on others. the number of users that I've seen rant about a game having downtime turning beta, doing server wipes, etc... They weren't complaining because they couldn't get enough bug reports in, but because they couldn't get into their Raid.

Because of pressure on various fronts, most of these games are released with insufficient server architecture, horrid bugs, and critical balance issues. This is the stuff that should be stomped out during beta, but it isn't. Beta isn't about testing, its about PR and hype. Wish it was some other way. If i was developing an MMO I'd want to disable users accounts that didn't file bug reports properly, but I know that doesn't do well for the PR side.

People feel entitled to their games, and even more entitled to a chance to play for free. As expected, its not uncommon for some big players in the game industry to give beta accounts to people who run big guilds, but don't necessarily put in bug reports.

Re:Halfway just for PR, Bad "beta testers"

[Bieeanda]

In several of the betas I've been in over the last few years, I've been faced with brief questionnaires and comment spaces at the end of every mission/quest/hallucination and quite often upon logging in or going to quit. If anything, their frequency in the early game tends to be annoyingly high, thanks to all of the quests designed to baby-step you through the interface.

But yes, most people who sign up for betas are absolutely fucking terrible at it. They have no QA training, and their posts on the official testing boards are usually 100% noise. It's even worse when they're dyed in the wool fanboys, because you'll never get useful criticism out of them.

Re:The Realm!

[Calmiche]

Sadly, it's not as dirty as it sounds.

The wrath was one of the rarest (early) swords in the game. You could enchant them with certain spells to increase strength and damage. However, the enchant spell had a chance of destroying the sword. I believe that a four enchant sword (4-banger) had a 80% chance of destroying the sword. For a time it was possible to get 5 enchants on a sword but the chance of getting 5 enchants was between 2% and 5%.

So, if you played a lot and were high level, you could find 2-3 Wraths a month on average. Then, you could burn through about 20 to 50 of them to get a 5 enchant sword.