Forgot your password?
typodupeerror
Networking Entertainment Games

How Demigod's Networking Problems Were Fixed 65

Posted by Soulskill
from the crunch-time dept.
The launch of Demigod was troubled by piracy and networking difficulties, which publisher Stardock worked quickly to correct. They've now released a documentary that gives a detailed look behind the scenes of diagnosing and fixing those problems. It includes meetings, interviews with the devs, and part of the bug-tracking process during a frenzied 108-hour work week.
This discussion has been archived. No new comments can be posted.

How Demigod's Networking Problems Were Fixed

Comments Filter:
  • Nice summary (Score:5, Insightful)

    by Jurily (900488) <jurily@@@gmail...com> on Sunday June 07, 2009 @06:16AM (#28240369)

    Am I supposed to watch half an hour of videos before they tell me what they actually did?

    Couldn't you just, you know, summarize it for us?

    • Re: (Score:2, Informative)

      by Anonymous Coward

      Don't bother, video 1 was some incredibly boring office meetings. video 2 I skipped to cut to the chase, and video 3 was home movie of the boss walking the halls annoying the ever living crap out of the developers actually trying to fix the problem. Again, boring!

    • by JustOK (667959)

      WTFV

    • by The Real Nem (793299) on Sunday June 07, 2009 @02:19PM (#28243319) Homepage

      I've summarized the technical reasons given by the three videos.

      Executive Summary:

      Seems like their peer-to-peer architecture exasperated otherwise common matchmaking and NAT transversal problems that should be expected and thoroughly tested when developing networked games.

      Video 1:

      • Summary: The game is peer-to-peer.

      Video 2:

      • Quote: "We're trying to figure out why users are being removed from the database, which keeps them from being connected, but their not really disconnected, but the server thinks they're disconnected."
      • Quote: "...and if you put a return to just ignore the disconnect message it works fine".

      Video 3:

      • Quote: We're working on a problem in which something, for some reason, keeps telling the database that manages people's records that they're disconnected and we don't know why.
      • Quote: So you're putting in the the debug messages?
      • Quote: Yep. Just putting in some debug messages to figure out where these ... what is, like ... wh what ... first of who is sending what, um these disconnects.
      • Quote: Right, okay, we'll be back.
      • Quote: "One of the fields in the database are backwards, or something, on IP addresses because there are so many IP addresses and ports and sockets and stuff that I don't know anything about."
      • Observation: They test a fix and it seems to work on several office PCs.
      • Summary: The description of the fix includes faster lobby connection time and visual changes to each players ping in the game lobby.
      • Paraphrase: How about in terms of the robustness of it, like, how much will people be able to connect?
      • Paraphrase: Their games are significantly more intelligent now. Um, there were are lot of crazy things that were going on before and we've been able to simplify a lot of them.
      • Paraphrase: What about proxy servers, do you think we need proxy servers now?
      • Paraphrase: No I don't think we need proxy servers. It might be good for backup; we were tossing around the thought, I think you brought this up, with routing some traffic if we can thorough the host. That may or may not be possible.
      • Paraphrase: Though we had to take care of the case if the host leaves the game we don't want the whole thing to fall apart.
      • Paraphrase: Right, right. But it might be a solution for players with low ping times who, you know, can't seem to connect to anybody, but, you know, want to get in the game.
      • Paraphrase: Right, that's true. Of course as a backup plan; so if the host left it would take out those people who have really bad connections. But based on what we're seeing, we think, this should handle symmetrical NATs now?
      • Paraphrase: What's that?
      • Paraphrase: Asymmetrical NATs; this should handle that now?
      • Paraphrase: Uh, it's going to handle it better, yeah.
      • Paraphrase: Well before it didn't handle it at all, so...
      • Paraphrase: Right before it didn't handle it at all, before...
      • Paraphrase: I mean if I have two IP addresses, will this work with it now?
      • Paraphrase: I don't know that if you have two IP addresses this will work. Uh, this is going to help alot with firewalls and nats, however, um, it does a better job of piercing through the firewall.
      • Paraphrase: Of course if you port forward to the right IP, then, from your router, then you're just set.
      • Paraphrase: Yeah, that's not an issue.
      • Paraphrase: So anyone who's technical will be fine, pretty much for sure, no matter on what kind of crazy connection they've got.
      • Paraphrase: And since we've spent more time improving the base systems in the program instead of doing one of the thing
  • Try testing (Score:2, Funny)

    by Anonymous Coward

    Maybe they should have made a documentary about their testing procedures instead, or would that be too short to fill even 10 minutes?

    • by uid7306m (830787)
      Yeah. Is it supposed to be cool to work too hard or is it supposed to be stupid? Seems to me like a properly planned and managed launch shouldn't be all that exciting. Or painful, whichever.
      • by Tiro (19535)
        Good point. Good development practices should have prevented a good portion of these issues, no matter which style.
        • Re: (Score:3, Funny)

          Yeah, cause if you use the right testing methodology you will never ever have any issues with your code. Netcraft confirms it!

  • Awesome... (Score:3, Funny)

    by Anonymous Coward on Sunday June 07, 2009 @06:22AM (#28240383)

    ...yet another Demigod Slashvertisement.

    • I want to work there, now.

    • Re: (Score:3, Informative)

      by XorNand (517466)
      Stardock is based in Plymouth, MI which is halfway between Detroit and Ann Arbor. Slashdot was originally founded in the same area (Dexter, MI IIRC). I live about 15 minutes north of Dexter and about 45 minutes from Plymouth. In case you haven't been following the news, Michigan doesn't have a whole lot to be proud of, economically-speaking. It's possible that the editors have a special fondness for Stardock. While independent game publishers are common in California, it's very rare to find a successful
    • Shouldn't they be trying to advertise on Warcraft 3? Or would Blizzard be to upset that they stole one of their user's creations?

  • by Manip (656104) on Sunday June 07, 2009 @06:38AM (#28240457)

    I've spoken with, and listened to, several game developers over the years and it just strikes me how different they have been to the guys on this. I'm watching the videos and it reminds me of my college days geeking it out with other geeky types and screwing around with code.

    Very different from the professional environment that you find in a lot of studios these days. They often seem to be run more like a Hollywood Movie or similar to commercial software companies and less like a college startup.

    But then again I've not played (or heard of) Demigod before this, and if they can produce fun games that's all that is really important. :)

    • But then again I've not played (or heard of) Demigod before this

      You must be new around here.

    • Re: (Score:3, Interesting)

      But then again I've not played (or heard of) Demigod before this

      That seems somewhat odd, as they've tried to get as much free advertising as they can by posting various development or piracy related stories to social news sites; some stories even got caught by the traditional media. It's not a bad marketing effort; people seem to be falling over themselves to get the game's name out there.

      I wonder how effective this type of advertising actually is though; does pointing out your mistakes and how you fixed them to a technical crowd win additional sales? Maybe that can b

      • by TheTurtlesMoves (1442727) on Sunday June 07, 2009 @08:21AM (#28240857)
        When a game has problems and they don't publish details everyone bemoans that fact that they are so closed about it. "Just be up front and honest about it, we understand there can be problems..." everyone says. Then when some does that everyone starts crying out " Cheap bastards are just trying to get free advertising... ".
        • Re: (Score:3, Insightful)

          When a game has problems and they don't publish details everyone bemoans that fact that they are so closed about it. "Just be up front and honest about it, we understand there can be problems..." everyone says. Then when some does that everyone starts crying out " Cheap bastards are just trying to get free advertising... ".

          Game developers often run into that problem and most (almost all?) of them have decided it's not worth being candid about their problems. It's interesting that Stardock have been able to twist their problems into publicity though, as over the five [slashdot.org] articles you'll find on Slashdot over the last couple of months about Demigod there are a lot of comments from people that've never heard of either before; you couldn't deny that these articles have given them a serious brand recognition boost, the only thing tha

          • Well I've never heard it before and I usually check the headlines every couple of days at least. But I only have the titles shown by default for the games section rather than the full summary, so I miss a lot of gaming stories. I'm now going to go have a google and see what the game actually is..

        • Re: (Score:3, Insightful)

          by Kaboom13 (235759)

          Maybe if their problems were actually interesting, and not the side effect of them being idiots without a clue how networking works, people would be more interested. Everything I've seen seems to revolve around them digging up a method of connecting players from early rts games, used primarily on LAN environments, not actually testing anything with real world equipment. The problems they had are to be expected.

          • by icedcool (446975)
            Its really easy to talk down to others mistakes.

            Stardock is a great company, ran really well. There one of the leaders on better drm practices... namely no drm. They boffed this one... it happens.
      • Well I have to admit to buying Demigod and would say that satisfied with it than other games I bought on the same service which have been a lot cheaper due to being released a while ago. Interestingly, the other games I am more satisfied with are the Supreme Commander titles which are by Gas Powered games who provided the engine (at least) for Demigod. I am unsure how satisfied I would have been had I bought the Supreme Commander titles when they were launched at full price.

        I haven't actually come across ma

        • by kfx (603703)

          Interestingly, the other games I am more satisfied with are the Supreme Commander titles which are by Gas Powered games who provided the engine (at least) for Demigod.

          GPG developed the entire game. Stardock published it and provided the Impulse Reactor overlay and the matchmaking/NAT-Traversal functions (via licensed code from RakNet initially, much of which has been replaced in the weeks since release). Once the NAT-T servers have connected the players together, they hand off to GPG's netcode for the actua

      • Re: (Score:3, Interesting)

        by dunezone (899268)
        Here is the thing about the advertising and no piracy protection.

        Demigod is a DOTA clone which was originally a Warcraft 3 map that is still widely played today and will more then likely be ported to Starcraft 2 if possible. So Demigod is competing against a game that already has a wide install base and is already cheap and will compete against a new game that will sell like bananers.
        Secondly, there is another game called League of Legends which is in the same style of Demigod that is coming out and is
        • Re: (Score:2, Interesting)

          by Anonymous Coward

          Your comment doesn't seem to have anything to do with advertising or piracy protection. What it does seem like is an attempt to use Demigod's limelight to get some publicity for LoL.

          Demigod's *retail* version ships without DRM because *all* Stardock games ship that way. It's notable that once you update the game through Impulse, though, the game is 'infected' with online activation DRM (but there are no limits on the number of activations from what I can tell).

          Furthermore, Demigod is not a DotA clone. It is

          • by Ifandbut (1328775)

            Demigod's *retail* version ships without DRM because *all* Stardock games ship that way. It's notable that once you update the game through Impulse, though, the game is 'infected' with online activation DRM (but there are no limits on the number of activations from what I can tell).

            There are no limits, just that the CD key is locked to your Impulse account so it acts alot like Steam in that you can download the game to any computer you want as long as you log in with your account. It is alot UNLIKE steam in that you can install the retail CD in a computer that is not connected to the internet and still play the game (whereas steam games force you to connect to the internet once).

  • Now TFA might show many things.
    What it doesn't: TFG just isn't fixed at all.

    There are unbefuckinglievable many ways to crash the game and even the servers crash at least once a week. The real server-crash-amount is probably only known to Stardock since everyone else doesn't try to play such a buggy game too often..

    But:
    Although I do think it's mostly a marketing thing, I still prefer Stardocks openness over "The problem is known and we're doing our best to fix it." type of answers which most other com
  • I mean, why it's so difficult to watch around and see what systems can be warez free.

    Offline gaming will always be pirated if anyone cares enough, but Internet gaming can be controlled. All you need is a serial number and registration. No DRM, Safedisk or any other bullshit. I mean World of Warcraft, has absolutely no CD protection, hell you can just copy the entire folder to a new PC and start to play, you don't even need installation. Yet they earn millions every day.
    • by jfim (1167051)

      I mean World of Warcraft, has absolutely no CD protection, hell you can just copy the entire folder to a new PC and start to play, you don't even need installation. Yet they earn millions every day.

      Why would they put copy protection on the game anyway, they make most of their money selling subscriptions, not retail boxes. Besides, even if someone copies the game and subscribes to WoW, they probably make more profit on one month of subscription fees than whatever trickles through the retail chain back to Blizzard.

      • by Kalriath (849904) *

        Why would they put copy protection on the game anyway, they make most of their money selling subscriptions, not retail boxes. Besides, even if someone copies the game and subscribes to WoW, they probably make more profit on one month of subscription fees than whatever trickles through the retail chain back to Blizzard.

        Yes, and they make more from people doing this too, since you need a retail CD key to upgrade an account from the trial version, and if you've copied a friend's game disk then you end up paying them $30 USD to generate you a CD key.

        So using World of Warcraft as an example is a bit like comparing apples to Ford Corollas.

  • poor management (Score:5, Insightful)

    by timmarhy (659436) on Sunday June 07, 2009 @07:52AM (#28240731)
    the ineptitude of their management can be seen in the 108 hour week. anything over a 12 hour day is wasted, and you NEED 1 day off a week minimum to recharge the batteries, otherwise you just find ways to waste time on the job.

    i've been there, i'm working 50 - 60 hour weeks and i achieve more now than i did in 90 hour weeks.

    • by juuri (7678)

      Way to post without bothering to invest even one moment in the source material.

    • Re: (Score:3, Interesting)

      by julesh (229690)

      the ineptitude of their management can be seen in the 108 hour week. anything over a 12 hour day is wasted, and you NEED 1 day off a week minimum to recharge the batteries, otherwise you just find ways to waste time on the job.

      i've been there, i'm working 50 - 60 hour weeks and i achieve more now than i did in 90 hour weeks.

      While I agree with what you're saying, it's only true long term. As long as they are _unusual_, long hours really can pull more work out of developers, without a necessarily huge drop i

      • by Stiletto (12066)

        Wrong. Any "crunch" time is a sign of a poorly managed project.

        You know what components need to be written, because they are all in the requirements. (You have requirements, don't you?)
        You know what functionality is required, because you have a functional spec. (You have a functional spec, don't you?)
        You know how all your components are going to behave, and how they're laid out, because you have a design spec (You have a design spec, don't you?)
        You know roughly how long it will take because you are properl

        • by julesh (229690)

          You know what components need to be written, because they are all in the requirements. (You have requirements, don't you?)

          Well, yes, but requirements can change. In this case, before the game was even released they suddenly found they needed to support about twice as many players on their servers as they were expecting in total. This exposed a number of bugs that might otherwise have remained hidden for long enough that they could be fixed one at a time without issue.

          You know what functionality is require

    • Re: (Score:3, Insightful)

      by SeeManRun (1040704)
      I just watched the entire video, and have no problem with what they did. Working in a game company, and any software company really, you learn that there is no substitute for real users. Once you get thousands of users online doing no deterministic things, your architecture crumbles. These guys did whatever they could to fix it, and for that I commend them. Hell, I was tempted to email the CEO and ask for a job until I learned they are in MI. I am very impressed with their efforts and the fact they doc
    • by antdude (79039)

      But they are a small company, not like EA, and had emergencies. So during emergencies like disasters, they can't do crazy hours?

    • by JimboFBX (1097277)
      At that rate, imagine what you could accomplish if you worked 1 hour a week.

      On a side note you should spend more time with your family.
  • by The_Quinn (748261) on Sunday June 07, 2009 @08:05AM (#28240785) Homepage
    A Government Solution:
    1. Pass regulations that don't allow buggy releases.
    2. Require dev studios to make games of each genre.
    3. Give special legal powers to developers to unionize, preventing firings, and forcing employers to deal with them.
    4. Give companies billions after they fail, and claim it was a failure of the free market.
    5. Appoint a Game Czar.
    6. Takeover the game companies, firing the company heads and appointing government stooges.
    7. Transfer ownership of the company to the government, give a minority stake to the developers.
    8. Make great games!
    • While your cynicism leaves me feeling sick, that is only because it is likely true and the alternatives aren't a lot better anyway. First actually funny joke I haven't been able to laugh at in a while.
  • by dunezone (899268) on Sunday June 07, 2009 @08:43AM (#28240943) Journal
    No to be a troll or anything but they haven't fixed multiplayer. They've released several patches over the past few weeks that have fixed issues but they tend to introduce more problems then fix.

    The game is good and I enjoy it a lot, but my god every time my friends and I decide to play we debate on rather we want to deal with all the connection issues. You spend more time waiting to get into a game lobby then playing the actual game.

    Like I said the game is good, and it has its share of issues even outside of multiplayer but to say its fixed is long from the truth.
    • by novex (515891)

      the game was ok, it suffered a variety of issues, but the networking was a flat out failure.

      i got my copy refunded after wasting 2 weeks trying to get anything resembling a $50 game worth of network gameplay.

      people shouldnt be charged to participate in alpha testing.

    • by pantherace (165052) on Sunday June 07, 2009 @01:02PM (#28242693)

      What's funny is when this article got posted on /., and last night, it was pretty much impossible to play online, due to something with their servers. For

      The game itself is good. Stardock's ImpulseReactor is bad. Impulse isn't so great. The engine isn't really to blame (some fundamental bugs though)
      I've played about 120 games online, and win about 50% of those.

      Here are some of the problems, having played it since pretty much the first day:
      In the initial release, they always tried to use NAT punchthrough. This includes when it wasn't needed. This put more load on their servers and made it slow to connect. Maybe piracy played some role, but given how it worked after it was 'fixed' I rather doubt piracy was the main problem, but more of an excuse.
      There are some ISPs (and routers) that mess with port numbering for UDP receive. This screws up the P2P network connection.
      Demigod/Supreme Commander speak a sandboxed lua in communication. Impulse sends and receives XML, and they wrote some translation layer in there, which was losing messages.
      Stardock implemented some proxies for fixing some problems, unfortunately the proxies are in my opinion, buggy. I've been sitting next to two people, and the same person connects to two of us fine, then proxied to the other. Uhhh, wtf? Additionally, they seem to have a tendency to crash, fucking up the game.
      According to Stardock, Stardock didn't write the NAT punchthrough, but licensed it from Raknet. Also, now Raknet of that denies it's used in Demigod. (And technically, it's not used in the engine, only the connection making, which is the real place where the BIG FUCKING PROBLEM is.)
      Impulse needs to be smacked with even the GNOME HIG guidelines, Apple's, even CDE's. Seriously, wtf? (I know it's trying to look like the latest office. There are so many people that have problems with the interface because of that. Unless you've seen Office a lot you won't recognize that the orb in the upper left is a menu, and even if you have seen office, you will likely miss it because it looks like an oversize decoration. Disclaimer: I don't like Office's new interface style in the first place.)
      Impluse's chat is a webpage, even though it's IRC, which requires IE's security settings to be set to default, it doesn't work if that's set higher.

      Demigod itself:
      Suffers from the problem of being designed with Microsoft's tools that have been focused on the Xbox, and using similar ideas. For example, running at the speed of the slowest isn't a problem when you have a homogeneous environment. PCs aren't homogeneous.
      Has some issues with the UI and communications due to all computers running the sim, and having to wait for every other computer's packets.
      Occasionally a desync (sim wasn't the same across all computers), which I have only seen about 3 times.
      Crashes occasionally, usually on connecting to a port, due to a UPNP call. (This might really belong under Impulse.)

      Plus sides:
      Less so now, but Stardock people are commonly on their IRC.
      If they are on, they generally try to be helpful, or such.
      The game is *very good*, especially on a LAN (or the internet when it works). Remarkably balanced in my opinion, for a newly released game. There are a couple of things which I think might need to be hit with a nerf bat, but they aren't very many.

      Oh, and while there is single player, it's not designed for it. For anyone wanting to play it, play a few games against the AI to understand the basic game, then DON'T PLAY AGAINST THE AI, if you ever intend to play humans. It teaches you bad habits. For example I played with friends against AIs a few games, getting something like a 33-1 K:D ratio against the 'hard' AI. The next games I played against humans, I got waxed.

      • Re: (Score:3, Informative)

        by atamido (1020905)

        Here are some of the problems, having played it since pretty much the first day: In the initial release, they always tried to use NAT punchthrough. This includes when it wasn't needed. This put more load on their servers and made it slow to connect. Maybe piracy played some role, but given how it worked after it was 'fixed' I rather doubt piracy was the main problem, but more of an excuse. There are some ISPs (and routers) that mess with port numbering for UDP receive. This screws up the P2P network connection.

        A big part of the problem was routers that performed Symmetric NAT. Basically, the router will randomize the outgoing port from the router, which is actually a sensible security feature. The problem was that the port would get randomized as something to their connection server, and then randomized to another port when connecting to another player. So that other player would attempt to connect back over a port that wasn't valid.

        What was funny about it is that you could forward specific ports to your compu

      • Re: (Score:3, Informative)

        by Rakthar (580956)

        Your comment of:

        >>Demigod itself suffers from the problem of being designed with Microsoft's tools that have been focused on the Xbox, and using similar ideas. For example, running at the speed of the slowest isn't a problem when you have a homogeneous environment. PCs aren't homogeneous.

        is wrong. They are using the Supreme Commander engine, which was designed for PCs in mind, and had the same issue. The problem is that SupCom and now Demigod are Peer 2 Peer, which means if one node is lagging and can

      • by Gnavpot (708731)

        In the initial release, they always tried to use NAT punchthrough. This includes when it wasn't needed.

        And if implemented like in Supreme Commander: The NAT punchthrough would in a lot of cases conflict with proper router forwarding of the game's listening port.

        Instead of trying to contact the port you forwarded in the router and had announced to the servers, other clients would ONLY try to contact the source port your router had used for your outbound connection to the server, hoping that your router wou

  • Demigods P2P connections are *NOT* fixed. This game is filled with desynchs, lag drops, the inability of two random players to connect and all sorts of various bugs. This does not include the extreme balance issues that are coming from HP stacking and classes that benefit from heavily from it (The UB) vs classes that are mana based and get waxed(TB). All in all i find it odd that a documentary of fixing something made it to slashdot of something that wasn't fixed. All in all i give this game a 9/10. Wit

Felson's Law: To steal ideas from one person is plagiarism; to steal from many is research.

Working...