Blizzard Offers Look Inside WoW At GDC 188
Yesterday morning at GDC Austin, Blizzard's J. Allen Brack and Frank Pearce took to the stage to finally give a peek inside the inner workings of World of Warcraft. Tipping the scales at around 4,600 people utilizing 20,000 computer systems and 1.3 petabytes of storage, Blizzard has created a raging behemoth. The Online Network services group alone has "data centers from Texas to Seoul, and monitor over 13,250 server blades, 75,000 cpu cores, and 112.5 terabytes of blade RAM. [Pearce] points out the picture of the GNOC (Global Network Operations Center) in their slideshow, a data core that even has televisions tuned to the weather stations. They use those to ensure that conditions of the data center are up to their standards; with only a staff of 68 people they ensure connectivity across the globe for the numerous WoW servers."
Story at 11... (Score:5, Funny)
Massive online game requires massive ammount of servers, bandwidth and people to maintain.....
Re: (Score:2)
ummm...how would I mod this...
Re:Story at 11... (Score:4, Funny)
With wild abandonment?
Re:Story at 11... (Score:4, Funny)
+5 He-Man And The Masters Of The Obvious!!1
Re: (Score:3, Interesting)
Re: (Score:2, Funny)
Re:Story at 11... (Score:5, Funny)
All you need is a post-it note saying "do not turn off!! SERVER!!!!"
Re: (Score:3, Insightful)
Then you find the one person who thinks unplugging the power strip to plug it into another outlet will not shut off the computer
Re: (Score:2, Funny)
You do not need such massive infrastructure to run a MMO, its just being wasteful about resources. Sure, the uptime could suffer a little bit and its possible you would sometimes run over the allowed bandwidth with Comcast, but you CAN run these things just fine on your living room behind the TV. If you want redundancy, you could have another server at your friends place.
Just because its "cool" and you have some money, it doesn't mean you have to waste it. Hell, even Slashdot runs just fine on CmdrTaco's mo
Re: (Score:2)
The best thing in this post is the "Informative" score.
Re: (Score:3, Insightful)
Re: (Score:2)
Re:Story at 11... (Score:5, Insightful)
...yes. The hard part of a Massively Multiplayer Online Game does in fact come from the Massively part.
Re: (Score:2)
Re: (Score:2)
Yes, implicit in my statement was that you had a number of players equivalent to WoW, along with the uptime, reliability, and latency of WoW. Perhaps I should have specifically said:
"You would be surprised at the number of people who believe that you can run a WoW server, supporting several thousand players with the same reliability and quality of the official server, on a spare box under your desk."
I should have expected pedantic replies on Slashdot. :)
Re: (Score:2, Funny)
Re: (Score:2)
Re: (Score:2)
One of the situations I had in mind during my post was the WoW launch, where irate players in queues demanded that Blizzard take some of their money down to Best Buy to buy new servers. Literally, and seriously.
Of course the same people kept asking why Blizzard didn't hire more programmers, even though they had job postings up for all the relevant positions. Apparently they expected that Blizzard could hire a lead server architect in a matter of weeks.
Most of these people claimed to be engineers or softwa
Re: (Score:2)
And a follow-up story at 12 - This just in: Computer nerds find massive amounts of servers, bandwidth, and people maintaining systems to be interesting. Some of them like hearing the details in success stories of large scale infrastructure. Even more so when the company itself is prominent or interesting.
Tune in tomorrow for our shocking revelation that some nerds also feel the need to mock these stories as un
Re:Story at 11... (Score:4, Interesting)
The interesting question with the amount of people working at blizzard is that will indie mmo's stand any change?
World of Warcraft utilizes 20,000 computer systems, 1.3 petabytes of storage, and more than 4600 people.
That's quite hard to compete with, and it only seems to be growing. Even other big MMO's have trouble competing with WoW, with Eve Online pretty much the only true competitor (and its more targeted for hardcore players)
Re: (Score:2)
Re: (Score:2)
Can someone explain this to me? I thought there were millions of WoW users? 4600 seems miniscule to me, especially when the same sentence says that they have 20,000 computer systems to serve these 4600 people... I don't understand.
Re: (Score:2)
4600 is the amount of people working on WoW, including the support staff and "two full-time lore historians, keepers of blizzard's past.". I'd like to have a work title "Warcraft Lore Historian"
Re: (Score:2)
As far as the millions- it takes 1 normal middle-high end computer to maintain 500 people logged in. If they have 20,000 servers, they can support their entire population (10 million) logging in at once (assuming they're evenly spread between servers).
Re: (Score:2)
Yes, WoW coders are actually from GetAFreelancer.com.
Re: (Score:2)
The tags say it all:
Blizzard games it! Wow, technology story.
Re: (Score:2)
The interesting thing to me as a possible future designer of a game of a similar scale, is how much and what is needed exactly. Because massive can differ quite a bit, from the power of a car ("a massive 800 PS") to the amount of atoms in the universe ("a massive 10E79 to 10E81 atoms").
In other words, it narrows it down a good bit. E.g. from the range of 800 to 10E81 servers. ^^
Re: (Score:2)
That was the case in 2005-2006, but that's not the case anymore. World breaking patches are the exception, not the rule, these days.
Whatever you need to tell yourself to help break the addiction, buddy.
Re: (Score:2)
They have 180k bugs in their tracking system. That's not the same thing. Considering their size and longevity, that's a remarkably low number ...
Well, Look at Their Monthly Revenue (Score:4, Insightful)
Re: (Score:2)
Did you read the article? They employ around 4000+ people on WoW alone.
Support staff:
Brack went on to talk about the customer support staff, a group with 2,056 game masters, 340 billing managers, and a host of other background staffers. These tireless staffers also work from locations around the world, ensuring that any local variations in culture (or the game) are respected.
Re: (Score:2)
I helped build a single datacenter that was a little more than 1/2 the total size of servers and staff they quote. It cost a little under 2 months worth of revenue by your numbers(*).
* I think the 10 million players is a highly massaged number. Sure there is a lot and they have probably had 10 million people sign up and pay for at least 1 month, but I don't think it's 10 million active paid accounts.
Re: (Score:3, Informative)
No, the 11.5 million number is accurate as at least up to 23Dec2008:
From: http://www.blizzard.com/us/press/081121.html [blizzard.com]
World of Warcraft's Subscriber Definition
World of Warcraft subscribers include individuals who have paid a subscription fee or have an active prepaid card to play World of Warcraft, as well as those who have purchased the game and are within their free month of access. Internet Game Room players who have accessed the game over the last thirty days are also counted as subscribers. The above d
Re:Well, Look at Their Monthly Revenue (Score:5, Informative)
Re:Well, Look at Their Monthly Revenue (Score:5, Funny)
Re: (Score:2, Insightful)
That's the first time in a long time that a Yakov Smirnov joke has made me do something other than want to slap people.
You get my vote...
Re: (Score:2)
Re: (Score:2)
I thought that the source of that meme was lost in the infinite data-stream that is the net
I prefer Nietsche as an original source, as in: "When you stare into the abyss the abyss stares back at you." That is, if the translation is correct.
Re: (Score:2)
Wait! if you just referenced the Simpson's ... that means ...
Re:Well, Look at Their Monthly Revenue (Score:5, Insightful)
"In Soviet Russia, the government controls the corporations."
is also pretty good, imho.
Re: (Score:2)
Maybe he modded from his sock puppet?
Re: (Score:2)
And so the labor and data rate tiering are different as well.
Re:Well, Look at Their Monthly Revenue (Score:5, Interesting)
Except ASIA is a large portion of the subscriber base (the 10 million number they like to quote a lot)and doesn't pay much per month at all. Blizzard licenses the game to ISPs and other partners that resell the game service as part of their offerings.
So that part of it IS known, and you should factor that into your equations. Monthly income off WoW is nowhere near $120 million.
Re: (Score:2)
I would imagine it is possible Malaysia also receives this version.
Perhaps. I do know that one of my guild leaders plays from Malaysia and we play on a US/Pacific TZ realm.
Re: (Score:2)
Re: (Score:2, Informative)
Also add in name changes ($10?), character re-customization ($20?), and server transfers ($25). Oh, and faction changes ($25?).
Re: (Score:2)
You figure it out. In China you have people dropping dead from overplaying. How many people in the US have you ever heard suffered that?
Should I Be Concerned... (Score:4, Interesting)
...that WoW servers are guarded and maintained better than DoD networks?
Re:Should I Be Concerned... (Score:5, Insightful)
Re: (Score:2)
Re: (Score:2)
Yet I can get one from Bizzard to protect my WoW account for $5.99???
They even gave them away at Blizzcon this year so I have an extra one...
Re: (Score:2)
Re: (Score:2)
More worrying about Chase is that they don't allow any non-alphanumeric characters in the password for the online banking site. I was kind-of shocked to find this out, because it suggests one, or both, of the following:
1) They don't salt/hash their passwords.
2) They don't used parameterized queries to protect against SQL injection.
Re: (Score:2)
The websites both have terrible usability issues as well but you would think that they would at least be a little concerned about security.
Re: (Score:2)
Which is weird since Rabobank is well known to be one of the first banks to offer electronic banking services over internet in the Netherlands, including devices from Vasco for security (one that accepts your bank card with chip/pin currently, and calculates a time based electronic signature for authentication and validating transactions.
Re: (Score:2)
What's that - en envelope without Windows?
Or an envelope mailed in a safe - wouldn't that be really pushing the envelope on what the mail system can handle easily?
Unless something got lost in translation, and the envelope really includes a condom - in which case, it's a "safe envelope", and not a "secure envelope."
Or it could be one of those Soviet Elect
Re:Should I Be Concerned... (Score:4, Informative)
Check out the terms of service on your bank account. You might be shocked to learn the bank isn't responsible for your financial losses. Often, they specifically exempt themselves from all responsibility relating to fraud, mistakes, and/or computer errors. If they cash a bad cheque, you are on the hook.
There is a reason why people that survived the Great Depression hide money under their mattresses.
Re: (Score:2)
Beyond that, when you're talking about bank failure (instead of fraud or stupidity), you've got the FDIC.
True, but a lot of people don't realize that FDIC only covers them in the case of bank failure -- even if the fraud comes from within the bank (embezzlement) and will probably eventually lead to the bank's failure. Just because the law allegedly protects you in case of fraud, that protection works at the speed of justice. If your account is one that the fraud affected, it could be frozen for years whil
Re: (Score:2)
That, and something about eggs and baskets.
Yeah I'd say more that, at least as far as protecting yourself from embezzlement goes. If your money isn't just frozen by the government, but gone, you're going to be one little guy in a long line of folks trying to get it back. There might not be any left by the time you get to the front of the line and the government is under no obligation to care. I had a friend who was in this situation... it suuuucks.
Multiple bank accounts ftw!
Re: (Score:2)
Uh, you should be alarmed and you should complain - All major (and come think of it even minor) banks offer authenticators and one time pads/scratch cards over here.
Re: (Score:2)
BoA does with their Safepass technology. You can buy a card authenticator for 20 bucks or use your cell phone which they send a code to.
Re: (Score:2)
I was alarmed when I was searching for a new bank that the major banks do not offer authenticators or usb dongles to use for online banking for normal consumers. Why can I protect my WoW account better than my bank account?
The solution seems obvious... convert all your cash to WoW gold to keep it safe.
On an unrelated subject, I have the best exchange rates between dollars and gold, and the speediest delivery! Money mailed to you within the hour!
Re: (Score:2)
Re: (Score:2)
Most DoD applications are pretty simple in comparison to a modern video game.
Um, no. The big difference is that the incoming crap on the screen will kill the operator and there will not be a quick run back to the corpse from a nearby graveyard.
All I read... (Score:5, Funny)
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
...need more rage?
Re:All I read... (Score:5, Informative)
I was told by someone at blizzard that they essentially implemented a fix across all battlegroups (which for those who don't know is a collection of realms at one data center) so you shouldn't see the error anymore. The problem was that each realm had a set amount of blades (something like 14?) for instances. Lower population realms didn't use hardly any of that capacity - whereas high population realms there wasn't enough. Well any good server admin knows you never can tell if a low population realm becomes a high population realm or visa versa so clearly you can't build these realms based off that alone - the app needs to scale accordingly.
The fix was that now all instances belong to a pool of servers now - which will eventually allow instance sharing across realms (that is - a party of players on different realms) once its switched on.
Re: (Score:2)
The fix was that now all instances belong to a pool of servers now - which will eventually allow instance sharing across realms (that is - a party of players on different realms) once its switched on.
That aligns with what I gathered. The biggest problem was that it took almost a month to roll the fix out to all the battlegroups. The solution was "known" for at least three weeks before my server saw any improvements.
Re: (Score:2)
You have to be kidding, this has been going on for a year on my server. They did nothing to fix it, just took our money. Their reputation in England is worse than toilet water at the moment. I dont care what their terms of service say, spending between 30 minutes and over two hours trying to get into instances for around a year is totally unacceptable.
So cancel your subscription? Dollars speak louder than words.
Re: (Score:2)
Re: (Score:2)
From what I hear from my WoW-playing friends, it wasn't ToC that broke it, it was the badge drop change that did it, people suddenly started chaining heroics en masse again. One of them was so happy because he managed to kit out his alt with both tier 8.5 tokens from badges in just 3 days.
Re: (Score:2)
His brain is probably still lagging a bit; possibly stuck in a mental Battle Ground.
"Only" 68 people? (Score:2, Insightful)
with only a staff of 68 people
How does it take 68 people to monitor that few servers, and most of them BLADES?!? The writers have apparently never worked in a large network environment (not that I'd expect that they would have, being writers and all). But seriously... that's not really that many servers for a large online service, it really shouldn't take that much work to keep it all running unless it's horribly designed.
Eh well, if they have the cash flow to retain that many warm bags of mostly water, m
Re: (Score:2)
If you were to assume that those 68 folks are at a central monitoring facility AND spread across multiple datacenters (Yes Timmy I said turn the power button off, pull out the bad blade, replace it with a good one) and you have 3 shifts (+1 Weekend) working 24/7 you might have 17 people on duty at any one time.
They also said responsible for connectivity, so a few of these guys are rebooting routers and load balancers, not just blades.
Re: (Score:2)
How does it take 68 people to monitor that few servers, and most of them BLADES?!?
The summary:
The Online Network services group alone has "data centers from Texas to Seoul, and monitor over 13,250 server blades, 75,000 cpu cores, and 112.5 terabytes of blade RAM. [Pearce] points out the picture of the GNOC (Global Network Operations Center) in their slideshow, a data core that even has televisions tuned to the weather stations. They use those to ensure that conditions of the data center are up to their standards; with only a staff of 68 people they ensure connectivity across the globe for the numerous WoW servers."
Is a bit confusing, to say the least. But it sounds like those 68 people are not monitoring a pile of blades in a single location. It sounds like those people are monitoring server scattered across the globe. Further, I doubt if all 68 of them work 24/7 - while the servers will need to be monitored nearly 24/7. So I'd assume there are various shifts through the day/week.
All things considered... 68 people doesn't sound like an absurd number to me.
Re: (Score:2)
Further, I doubt if all 68 of them work 24/7
Only the ones in Seoul work 24/7.
Re: (Score:2)
It seems about right to me.
Re: (Score:2)
Small programming dept (Score:3, Interesting)
The programming department currently consists of 32 people, and envelopes systems, tools, gameplay, server technologies, and UI.
I know adding more developers can slow down production in the short term, but 5 years on I would think they would have been able to scale their programming staff up a bit more by now. New ui elements (gear manager, quest helper, even voice chat) have tended to be late and light on features, so thats one area I would think could benifit from more bodies in the future.
Re: (Score:2)
I think you need to (re)read 'The Mythical man month', more people does not equal more productivity. Not to mention that some features have been 'late' not because they were difficult to code, but likely because they didn't feel they were necessary yet. Not to mention (2) that every new thing creates more QA, and that the game needs to remain accessible for new players, not only for people that have been there for years.
Re: (Score:2)
At 5.5 mil LOC, you'd be lucky if you could even *find* the source of one bug a day.
There is a balancing act involved. I remember an old chestnut that said something to the effect of "A product is good enough to release when it works for 85% of the people trying to use it". Go above 85%, and you start getting diminishing returns on cost. Could they fix every single little glitch? Sure. From a business perspective, is it worth the price? No way.
The whole point of the Mythical Man Month is not that more coder
No... (Score:2)
It can hurt in the long term and the short term. You get too many bright people on a project and it takes forever to reach consensus on entirely too much. Particularly if each is passionate about the entire product beyond their small piece and have strong opinions/vision about how it should be done. Especially if they are users/fans of their own software.
Re: (Score:2)
MMOs have one two weaknesses:
The first is that every new feature needs to be tested for software regressions thoroughly. Not to mention unexpected code or content interactions that players can use to gain a disproportionate disadvantage.
For example, there was a bug with one game where if one knew the constants that skills were named, one could try manually type in the skill name and learn it, even if that skill was never a part of that class. So you would have warriors running around who could open a figh
Re: (Score:2)
Why would scaling up the programming staff fix those problems? How do you they aren't design problems from elsewhere in the hierarchy?
Pictures? (Score:2)
So wheres the pictures?
Brack?? (Score:2)
Production Director Brack and Game Director Tom Chilton are...
All hail Brack!!!!
Zorak Rules (Score:2)
Forget Brak. Zorak kicks major ass.
And don't knock him just because he is a virgin, and has never been eaten by a female. The same is true for the majority of Slashdotters too!
Re: (Score:2)
DON'T TOUCH ME
Bosses (Score:5, Insightful)
Re: (Score:3, Funny)
Apparently the programmer's boss is also a programmer, the artists boss an artist and they are expected to work together. So so SOOOO much better than the bureaucrats most of us get stuck with.
Yeah. You say that now. Then you'll get a job where your boss is a programmer, and it'll be like "Why haven't you finished that task yet? I could have done that in 2 hours, and you've been 6...", and no matter how much you argue about how long such a task takes, you'll never win, because he'll _know_ exactly where [xkcd.com]
Re: (Score:2)
I once worked in a group where the previous supervisor was a coder. He was known for coding anything he thought his employees did too slowly or poorly.
Re: (Score:2)
Of course these low level managers have much more stress than people who work for them. Not only do they do the typical manager bureaucratic nonsense, but after attending 6 hours of meetings a day they still have another 6 hours of pro
Imagine what these guys are like... (Score:4, Funny)
I wonder what those two guys are like. I'm pretty sure they must be nerds of EPIC proportions. And I don't mean that in a mean way, I'm just sayin....
Re: (Score:2)
Ob. ctrl-alt-del (Score:4, Funny)
How it REALLY happens behind closed doors:
http://ctrlaltdel-online.com/comic.php?d=20090916 [ctrlaltdel-online.com]
How? (Score:2)
Weather Stations (Score:2, Funny)
a data core that even has televisions tuned to the weather stations.
That the night shift promptly changes to Family Guy at 8 PM when everyone else goes home.
Re: (Score:2)
Not much detailing about their data centers. It seemed more of a Blizzard PR piece than anything else, and without technical details, this is just another Blizzard ad.
"The PR and community teams were Brack's next focus, the groups responsible for public interaction. The PR team has helped to ensure some 10,000 articles have been written about World of Warcraft"
Job well done, PR team!