Slashdot is powered by your submissions, so send in your scoop

 


Forgot your password?
Close
typodupeerror
×
Games IT Technology

Blizzard Offers Look Inside WoW At GDC 188

Posted by ScuttleMonkey from the belly-of-the-beast dept.
Yesterday morning at GDC Austin, Blizzard's J. Allen Brack and Frank Pearce took to the stage to finally give a peek inside the inner workings of World of Warcraft. Tipping the scales at around 4,600 people utilizing 20,000 computer systems and 1.3 petabytes of storage, Blizzard has created a raging behemoth. The Online Network services group alone has "data centers from Texas to Seoul, and monitor over 13,250 server blades, 75,000 cpu cores, and 112.5 terabytes of blade RAM. [Pearce] points out the picture of the GNOC (Global Network Operations Center) in their slideshow, a data core that even has televisions tuned to the weather stations. They use those to ensure that conditions of the data center are up to their standards; with only a staff of 68 people they ensure connectivity across the globe for the numerous WoW servers."
This discussion has been archived. No new comments can be posted.

Blizzard Offers Look Inside WoW at GDC More

Blizzard Offers Look Inside WoW at GDC

Comments Filter:

  • Story at 11... (Score:5, Funny)

    by CRiMSON ( 3495 ) <<gro.elbakaepsnu> <ta> <nosmirc>> on Friday September 18, 2009 @01:02PM (#29468223) Homepage

    Massive online game requires massive ammount of servers, bandwidth and people to maintain.....

    • ummm...how would I mod this...

    • Re: (Score:3, Interesting)

      by Rycross ( 836649 )
      You'd be surprised at the number of people that think you can run something like a WoW server on a spare box underneath someone's desk.

      • Re: (Score:2, Funny)

        by NYMeatball ( 1635689 )
        You wouldn't be surprised at the number of people who DO try to run application servers underneath their own desk in the corporate world.

      • Re: (Score:2, Funny)

        by sopssa ( 1498795 ) *

        You do not need such massive infrastructure to run a MMO, its just being wasteful about resources. Sure, the uptime could suffer a little bit and its possible you would sometimes run over the allowed bandwidth with Comcast, but you CAN run these things just fine on your living room behind the TV. If you want redundancy, you could have another server at your friends place.

        Just because its "cool" and you have some money, it doesn't mean you have to waste it. Hell, even Slashdot runs just fine on CmdrTaco's mo

      • Re: (Score:2)

        by Korin43 ( 881732 )
        I used to run a WoW server on normal slow computer. It worked fine. I'd imagine the hard part is getting decent uptime and speeds with thousands of players.
      • You can, actually, so long as no more than about 500 people are on at a time. That is the the average load for a single machine, with medium-high specs. If you get a cluster running, you can do a normal sized wow server. Just look up ascentemu.

        • Re: (Score:2)

          by Rycross ( 836649 )

          Yes, implicit in my statement was that you had a number of players equivalent to WoW, along with the uptime, reliability, and latency of WoW. Perhaps I should have specifically said:

          "You would be surprised at the number of people who believe that you can run a WoW server, supporting several thousand players with the same reliability and quality of the official server, on a spare box under your desk."

          I should have expected pedantic replies on Slashdot. :)

      • I'm not surprised. I often hear players in some games complain how they can do a better job than the devs, and that if the "server" is down why don't they just get a new computer or replace the ram.

        • Re: (Score:2)

          by Rycross ( 836649 )

          One of the situations I had in mind during my post was the WoW launch, where irate players in queues demanded that Blizzard take some of their money down to Best Buy to buy new servers. Literally, and seriously.

          Of course the same people kept asking why Blizzard didn't hire more programmers, even though they had job postings up for all the relevant positions. Apparently they expected that Blizzard could hire a lead server architect in a matter of weeks.

          Most of these people claimed to be engineers or softwa

    • Massive online game requires massive ammount of servers, bandwidth and people to maintain.....

      And a follow-up story at 12 - This just in: Computer nerds find massive amounts of servers, bandwidth, and people maintaining systems to be interesting. Some of them like hearing the details in success stories of large scale infrastructure. Even more so when the company itself is prominent or interesting.

      Tune in tomorrow for our shocking revelation that some nerds also feel the need to mock these stories as un

    • Tipping the scales at around 4600 people utilizing 20,000 computer systems

      Can someone explain this to me? I thought there were millions of WoW users? 4600 seems miniscule to me, especially when the same sentence says that they have 20,000 computer systems to serve these 4600 people... I don't understand.

      • Re: (Score:2)

        by sopssa ( 1498795 ) *

        4600 is the amount of people working on WoW, including the support staff and "two full-time lore historians, keepers of blizzard's past.". I'd like to have a work title "Warcraft Lore Historian"

      • I believe they're referring to the number of blizzard employees involved in the area.
        As far as the millions- it takes 1 normal middle-high end computer to maintain 500 people logged in. If they have 20,000 servers, they can support their entire population (10 million) logging in at once (assuming they're evenly spread between servers).

    • The tags say it all:

      Blizzard games it! Wow, technology story.

    • The interesting thing to me as a possible future designer of a game of a similar scale, is how much and what is needed exactly. Because massive can differ quite a bit, from the power of a car ("a massive 800 PS") to the amount of atoms in the universe ("a massive 10E79 to 10E81 atoms").

      In other words, it narrows it down a good bit. E.g. from the range of 800 to 10E81 servers. ^^

  • Well, Look at Their Monthly Revenue (Score:4, Insightful)

    by eldavojohn ( 898314 ) * <eldavojohn@noSpAM.gmail.com> on Friday September 18, 2009 @01:07PM (#29468291) Journal
    Let's say they have 10 million active subscribers world wide and that each of them pays $12 a month. Wouldn't you expect that sort of protection and insane support on something generating $120 million in revenue for you a month? I would. I wouldn't be surprised if there's a whole lot more to it that we don't know about and never will.

    • Re: (Score:2)

      by sopssa ( 1498795 ) *

      Did you read the article? They employ around 4000+ people on WoW alone.

      Support staff:

      Brack went on to talk about the customer support staff, a group with 2,056 game masters, 340 billing managers, and a host of other background staffers. These tireless staffers also work from locations around the world, ensuring that any local variations in culture (or the game) are respected.

    • I helped build a single datacenter that was a little more than 1/2 the total size of servers and staff they quote. It cost a little under 2 months worth of revenue by your numbers(*).

      * I think the 10 million players is a highly massaged number. Sure there is a lot and they have probably had 10 million people sign up and pay for at least 1 month, but I don't think it's 10 million active paid accounts.

      • Re: (Score:3, Informative)

        by Trolan ( 42526 )

        No, the 11.5 million number is accurate as at least up to 23Dec2008:

        From: http://www.blizzard.com/us/press/081121.html [blizzard.com]

        World of Warcraft's Subscriber Definition

        World of Warcraft subscribers include individuals who have paid a subscription fee or have an active prepaid card to play World of Warcraft, as well as those who have purchased the game and are within their free month of access. Internet Game Room players who have accessed the game over the last thirty days are also counted as subscribers. The above d

  • Should I Be Concerned... (Score:4, Interesting)

    by Logical Zebra ( 1423045 ) on Friday September 18, 2009 @01:13PM (#29468421)

    ...that WoW servers are guarded and maintained better than DoD networks?

    • Re:Should I Be Concerned... (Score:5, Insightful)

      by drexlor ( 1314419 ) on Friday September 18, 2009 @01:45PM (#29468851)
      I was alarmed when I was searching for a new bank that the major banks do not offer authenticators or usb dongles to use for online banking for normal consumers. Why can I protect my WoW account better than my bank account?
      • FYI, E*Trade offers online banking and will let you use an RSA dongle for authentication. They'll charge you for it, mind you, unless you have a boatload of money parked with them, but it's available.

        • Yet I can get one from Bizzard to protect my WoW account for $5.99???

          They even gave them away at Blizzcon this year so I have an extra one...

      • Re:Should I Be Concerned... (Score:4, Informative)

        by Cassini2 ( 956052 ) on Friday September 18, 2009 @02:18PM (#29469349)

        Why can I protect my WoW account better than my bank account?

        Check out the terms of service on your bank account. You might be shocked to learn the bank isn't responsible for your financial losses. Often, they specifically exempt themselves from all responsibility relating to fraud, mistakes, and/or computer errors. If they cash a bad cheque, you are on the hook.

        There is a reason why people that survived the Great Depression hide money under their mattresses.

      • Re: (Score:2)

        by tero ( 39203 )

        Uh, you should be alarmed and you should complain - All major (and come think of it even minor) banks offer authenticators and one time pads/scratch cards over here.

      • BoA does with their Safepass technology. You can buy a card authenticator for 20 bucks or use your cell phone which they send a code to.

      • I was alarmed when I was searching for a new bank that the major banks do not offer authenticators or usb dongles to use for online banking for normal consumers. Why can I protect my WoW account better than my bank account?

        The solution seems obvious... convert all your cash to WoW gold to keep it safe.

        On an unrelated subject, I have the best exchange rates between dollars and gold, and the speediest delivery! Money mailed to you within the hour!

      • Re: (Score:2)

        by afidel ( 530433 )
        Bank of America and PNC both offer 2 factor authentication.

  • All I read... (Score:5, Funny)

    by lazorz ( 1544583 ) on Friday September 18, 2009 @01:16PM (#29468467)
    Additional instances cannot be launched.
    • Must construct additional servers!

    • Re:All I read... (Score:5, Informative)

      by Skuld-Chan ( 302449 ) on Friday September 18, 2009 @02:14PM (#29469279)

      I was told by someone at blizzard that they essentially implemented a fix across all battlegroups (which for those who don't know is a collection of realms at one data center) so you shouldn't see the error anymore. The problem was that each realm had a set amount of blades (something like 14?) for instances. Lower population realms didn't use hardly any of that capacity - whereas high population realms there wasn't enough. Well any good server admin knows you never can tell if a low population realm becomes a high population realm or visa versa so clearly you can't build these realms based off that alone - the app needs to scale accordingly.

      The fix was that now all instances belong to a pool of servers now - which will eventually allow instance sharing across realms (that is - a party of players on different realms) once its switched on.

      • The fix was that now all instances belong to a pool of servers now - which will eventually allow instance sharing across realms (that is - a party of players on different realms) once its switched on.

        That aligns with what I gathered. The biggest problem was that it took almost a month to roll the fix out to all the battlegroups. The solution was "known" for at least three weeks before my server saw any improvements.

  • "Only" 68 people? (Score:2, Insightful)

    by FuegoFuerte ( 247200 )

    with only a staff of 68 people

    How does it take 68 people to monitor that few servers, and most of them BLADES?!? The writers have apparently never worked in a large network environment (not that I'd expect that they would have, being writers and all). But seriously... that's not really that many servers for a large online service, it really shouldn't take that much work to keep it all running unless it's horribly designed.

    Eh well, if they have the cash flow to retain that many warm bags of mostly water, m

    • If you were to assume that those 68 folks are at a central monitoring facility AND spread across multiple datacenters (Yes Timmy I said turn the power button off, pull out the bad blade, replace it with a good one) and you have 3 shifts (+1 Weekend) working 24/7 you might have 17 people on duty at any one time.

      They also said responsible for connectivity, so a few of these guys are rebooting routers and load balancers, not just blades.

    • How does it take 68 people to monitor that few servers, and most of them BLADES?!?

      The summary:

      The Online Network services group alone has "data centers from Texas to Seoul, and monitor over 13,250 server blades, 75,000 cpu cores, and 112.5 terabytes of blade RAM. [Pearce] points out the picture of the GNOC (Global Network Operations Center) in their slideshow, a data core that even has televisions tuned to the weather stations. They use those to ensure that conditions of the data center are up to their standards; with only a staff of 68 people they ensure connectivity across the globe for the numerous WoW servers."

      Is a bit confusing, to say the least. But it sounds like those 68 people are not monitoring a pile of blades in a single location. It sounds like those people are monitoring server scattered across the globe. Further, I doubt if all 68 of them work 24/7 - while the servers will need to be monitored nearly 24/7. So I'd assume there are various shifts through the day/week.

      All things considered... 68 people doesn't sound like an absurd number to me.

    • It seems about right to me.

    • No, 68 for IT operations across multiple datacenters sounds about right. 2 shifts during the week, 2 during the weekend (at least), and a minimum of 2 operators per shift makes for 8 operators needed for a 24/7 environment. Looking around online, it looks like they have 4 US datacenters [wowwiki.com], 1 Latin America datacenter [wowwiki.com], 1 European datacenter [wowwiki.com], 1 Korean datacenter [wowwiki.com], 4 Chinese datacenters [wowwiki.com], and 1 Taiwanese datacenter [wowwiki.com]. With a support staff of 68 and 12 DCs, they have an average of 5 and 2/3s operators per DC. They act

  • Small programming dept (Score:3, Interesting)

    by obi1one ( 524241 ) on Friday September 18, 2009 @01:20PM (#29468525)

    The programming department currently consists of 32 people, and envelopes systems, tools, gameplay, server technologies, and UI.

    I know adding more developers can slow down production in the short term, but 5 years on I would think they would have been able to scale their programming staff up a bit more by now. New ui elements (gear manager, quest helper, even voice chat) have tended to be late and light on features, so thats one area I would think could benifit from more bodies in the future.

    • I think you need to (re)read 'The Mythical man month', more people does not equal more productivity. Not to mention that some features have been 'late' not because they were difficult to code, but likely because they didn't feel they were necessary yet. Not to mention (2) that every new thing creates more QA, and that the game needs to remain accessible for new players, not only for people that have been there for years.

    • No... (Score:2)

      by Junta ( 36770 )

      It can hurt in the long term and the short term. You get too many bright people on a project and it takes forever to reach consensus on entirely too much. Particularly if each is passionate about the entire product beyond their small piece and have strong opinions/vision about how it should be done. Especially if they are users/fans of their own software.

    • Re: (Score:2)

      by mlts ( 1038732 ) *

      MMOs have one two weaknesses:

      The first is that every new feature needs to be tested for software regressions thoroughly. Not to mention unexpected code or content interactions that players can use to gain a disproportionate disadvantage.

      For example, there was a bug with one game where if one knew the constants that skills were named, one could try manually type in the skill name and learn it, even if that skill was never a part of that class. So you would have warriors running around who could open a figh

    • Why would scaling up the programming staff fix those problems? How do you they aren't design problems from elsewhere in the hierarchy?

  • So wheres the pictures?

  • FTA:

    Production Director Brack and Game Director Tom Chilton are...

    All hail Brack!!!!

  • Bosses (Score:5, Insightful)

    by Idiomatick ( 976696 ) on Friday September 18, 2009 @01:46PM (#29468875)
    Apparently the programmer's boss is also a programmer, the artists boss an artist and they are expected to work together. So so SOOOO much better than the bureaucrats most of us get stuck with.

    • Re: (Score:3, Funny)

      by julesh ( 229690 )

      Apparently the programmer's boss is also a programmer, the artists boss an artist and they are expected to work together. So so SOOOO much better than the bureaucrats most of us get stuck with.

      Yeah. You say that now. Then you'll get a job where your boss is a programmer, and it'll be like "Why haven't you finished that task yet? I could have done that in 2 hours, and you've been 6...", and no matter how much you argue about how long such a task takes, you'll never win, because he'll _know_ exactly where [xkcd.com]

      • Re: (Score:2)

        by Dog-Cow ( 21281 )

        I once worked in a group where the previous supervisor was a coder. He was known for coding anything he thought his employees did too slowly or poorly.

    • I've had programmers and engineers as managers for more than a decade. I always thought the days of working directly for a generic manager were over. Even my manager's managers tend to be knowledgeable about things, or were engineers. Usually true all the way up to the CTO

      Of course these low level managers have much more stress than people who work for them. Not only do they do the typical manager bureaucratic nonsense, but after attending 6 hours of meetings a day they still have another 6 hours of pro

  • Imagine what these guys are like... (Score:4, Funny)

    by dan_sdot ( 721837 ) on Friday September 18, 2009 @02:43PM (#29469677)

    The creative development team is the hub for the company's history. They have two full-time lore historians, keepers of blizzard's past. They are the liaisons with the novelists, work to create shared art resources, act as an archive for every piece of art that's been created for Blizzard Entertainment, and currently maintain 100,000 art assets.

    I wonder what those two guys are like. I'm pretty sure they must be nerds of EPIC proportions. And I don't mean that in a mean way, I'm just sayin....

  • Ob. ctrl-alt-del (Score:4, Funny)

    by DarthVain ( 724186 ) on Friday September 18, 2009 @04:17PM (#29470915)

    How it REALLY happens behind closed doors:

    http://ctrlaltdel-online.com/comic.php?d=20090916 [ctrlaltdel-online.com]

  • Interesting part in the article : the programming team only has slightly over 30 developers. The QA department has 6 times that. One would think that a product so complex would need more coders.

  • a data core that even has televisions tuned to the weather stations.

    That the night shift promptly changes to Family Guy at 8 PM when everyone else goes home.

Slashdot Top Deals

Beware of Programmers who carry screwdrivers. -- Leonard Brandwein

Close