Slashdot is powered by your submissions, so send in your scoop

 



Forgot your password?
typodupeerror
×
Games

EVE Online Battle Breaks Records (And Servers) 308

captainktainer writes "In one of the largest tests of EVE Online's new player sovereignty system in the Dominion expansion pack, a fleet of ships attempting to retake a lost star system was effectively annihilated amidst controversy. Defenders IT Alliance, a coalition succeeding the infamous Band of Brothers alliance (whose disbanding was covered in a previous story), effectively annihilated the enemy fleet, destroying thousands of dollars' worth of in-game assets. A representative of the alliance claimed to have destroyed a minimum of four, possibly five or more of the game's most expensive and powerful ship class, known as Titans. Both official and unofficial forums are filled with debate about whether the one-sided battle was due to difference in player skill or the well-known network failures after the release of the expansion. One of the attackers, a member of the GoonSwarm alliance, claims that because of bad coding, 'Only 5% of [the attackers] loaded,' meaning that lag prevented the attackers from using their ships, even as the defenders were able to destroy those ships unopposed. Even members of the victorious IT Alliance expressed disappointment at the outcome of the battle. CCP, EVE Online's publisher, has recently acknowledged poor network performance, especially in the advertised 'large fleet battles' that Dominion was supposed to encourage, and has asked players to help them stress test their code on Tuesday. Despite the admitted network failure, leaders of the attacking force do not expect CCP to replace lost ships, claiming that it was their own fault for not accounting for server failures. The incident raises questions about CCP's ability to cope with the increased network use associated with their rapid growth in subscriptions."
This discussion has been archived. No new comments can be posted.

EVE Online Battle Breaks Records (And Servers)

Comments Filter:
  • Kinda Cool (Score:5, Interesting)

    by MarkvW ( 1037596 ) on Tuesday January 05, 2010 @01:16AM (#30651784)

    I still don't think I'll sign over my credit card to a MM online game, but a game that lets you destroy THOUSANDS of dollars of stuff that other people value for the sheer malicious joy . . . well, that's perversely COOL!

    • Re: (Score:3, Funny)

      by TheLink ( 130905 )
      Some people already do that in the financial world.

      Oh yeah, they do treat it as a game.
    • by syousef ( 465911 )

      I still don't think I'll sign over my credit card to a MM online game, but a game that lets you destroy THOUSANDS of dollars of stuff that other people value for the sheer malicious joy . . . well, that's perversely COOL!

      Destroy thousands of dollars of stuff? You mean donate it to CCP games don't you?

    • Re: (Score:3, Interesting)

      by jollyreaper ( 513215 )

      I still don't think I'll sign over my credit card to a MM online game, but a game that lets you destroy THOUSANDS of dollars of stuff that other people value for the sheer malicious joy . . . well, that's perversely COOL!

      I noodled about in EVE years ago. It's got the harshest cost factor of any MMO out there. It can take weeks of grinding to get a good ship and you can lose it in seconds if you're not careful. You have many, many hours spread between the ship you buy and your pilot and the pilot has both XP and implants that boost stats. You can buy clone insurance for the pilot but implants are always lost upon death.

      There are a lot of chinese farmers in the game as well as OCD no-life guys who amass virtual fortunes. You

  • by GammaKitsune ( 826576 ) on Tuesday January 05, 2010 @01:16AM (#30651788)
    Only in EVE would the players decide that network failures are a factor they should take into consideration.
    • by Dan667 ( 564390 )
      EVE has an actual functioning economy and the main story is actually driven by the players themselves. You are encouraged to use everything at your disposal even play multiple accounts so it only makes sense network issues would become part of the game.
    • by cbhacking ( 979169 ) <been_out_cruisin ... nospAM.yahoo.com> on Tuesday January 05, 2010 @02:20AM (#30652126) Homepage Journal

      Only in EVE would you try and have thousands of players meet in a single location to fight.

      From the sound of it, the number of people who were in that particular star system (or trying to get in) exceeds the number of players on many WOW instances. Yet, all in all, that was probably at most a few percent of the players online at the time, and they're all connected to the same game world.

      That said, a single star system on EVE is hosted by a single physical server. Less-used systems can be grouped together to save on hardware, but for a big fight like this CCP fires up their most powerful hardware and puts the relevant system(s) on dedicated servers. While they're getting good at this - a few years ago 200 ships was a big fight, these days it's a common occurrence - it's still going to be an awful strain on the server to support that many players in combat. In a situation like that, the players need to take the limitations of computer hardware into account, and plan accordingly.

      • by Opportunist ( 166417 ) on Tuesday January 05, 2010 @02:35AM (#30652214)

        In a situation like that, the players need to take the limitations of computer hardware into account, and use it to gain an edge.

        It's EvE, after all.

      • by Anonymous Coward on Tuesday January 05, 2010 @04:00AM (#30652656)

        In this particular case, there were 1340 people in the same star system at peak. A star system is handled by a single server, due to code limitations - CCP have stated that they want to improve the code so that a single system can be managed by multiple servers, but they are not yet there. About a year ago, this number of people fighting in the same system would've been next to impossible. CCP later improved their code to be able to handle fleet fights of 1500+ players with reasonable responsiveness (IE, lagged but playable). Due to the fact that eve is more of a tactical simulator than an action sim, lag is not always such a big issue. A five or ten second weapons activation lag is actually playable in a fashion, unlike many other games.

        Now, keep in mind that this is actually 1340 players fighting on a single physical server, with upwards of 50,000 players logged in at the same time on the single world shard of eve. Compare this to many other MMOs, where you may have 1340 players total on a shard cluster.

        Now, EVE is not designed for 1340 players. The fact that that many players are able to play in a single solar system at all is a testament to the sandbox nature of eve, where the developers have decided to try to avoid hard limits as much as possible (IE, no 25-man raids or maximum players on the server), but instead allow the players to use as much as they can and want. This obviously results in situations where the servers cannot cope, which is a known problem with fleet fights. CCP's response has traditionally been "Yes, we allow you to do this, but be aware of the potential consequences - we won't reimburse you for lag or poor server performance". The alternative would've been hard limits on the number of people on a node, which would've favoured those who made it in first, with the most people on their side - there are no defined sides in eve, so you cannot for instance let in 100 red and 100 blue.

        The battle that the OP refers to was one of those cases. It was well known that server performance was unreliable after the Dominion patch. In many cases, this would prevent fleet fights from occurring, and when they did occur, they were often one-sided massacres. Knowing this, and despite being warned by their allies numerous times, the opposing force still decided to enter the system. Not only that, but they also decided to jump in at the same time, instead of staggering their jumpins - something that has been proven to reduce lag and avoid people getting stuck in loading - or jumping in to different "grids". In fact, leaked logs indicate that they did this knowingly with the intent of crashing an already overloaded node, so that they'd be at an even footing when the server came back up.

        In the end, this backfired and they lost their entire fleet as a result.

        Once again, CCP allows fleetfights with no hard limit on the number of participants, but their stance is "Yes, we allow you to do this, but be aware of the potential consequences - we won't reimburse you for lag or poor server performance".

        • Re: (Score:3, Interesting)

          by Shinobi ( 19308 )

          Actually, a year ago, there were even larger battles that were functional. The problem is CCP apparently not having synched development trees. So for example, every other expansion, drone AI gets old bugs back, that were fixed in the release in-between. The Bloodlines expansion was the worst in terms of that though

        • by Weezul ( 52464 ) on Tuesday January 05, 2010 @08:23AM (#30653896)

          True, but CCP also has a long history of favoring the Band of Brothers. People can quite reasonably accuse CCP of continuing to favor BoB's playing style here.

          A fleetfight should simply not become unbalanced. If players are lagging out, they should cull players evenly from each side, ideally offering non-culled players the option to give their slot to a culled player.

          • by zerocool^ ( 112121 ) on Tuesday January 05, 2010 @03:20PM (#30659802) Homepage Journal

            CCP's favoritism of BoB is something that gets trotted out every time they accomplish anything. It's years in the past.

            In this engagement, BoB (IT) won. The people who jumped in were dumb. They knew this might happen.

            In a previous engagement (check the corporation alliance and org forum for post by SK Rooster), BoB/IT jumped in to someone else and lost 40 dreadnoughts. Favoritism? Not so much. Whenever BoB loses, it's cause they suck, whenever they win, it's because they are getting help from the Developers and GM's. Right?

      • by brkello ( 642429 )
        No, it wasn't more than a WoW shard. It is also a much simpler game. And they continue to not be able to handle large scale battles when advertising the opposite.

        While the above is fact, my opinion is it is an extremely boring game except for the people who can dedicate a lot of time to it. It is quite tedious to play as someone who only plays like 8 hours a week. I have tried to get in to it a few times and just die of boredom after a week or two.
    • Re: (Score:3, Insightful)

      by Ephemeriis ( 315124 )

      Only in EVE would the players decide that network failures are a factor they should take into consideration.

      EVE is a fairly intense game.

      It can take you literally years to train enough skills to fly certain ships. It can take weeks to manufacture a piece of equipment. It can take months to recover from a loss.

      Yes, of course, it's only pixels... But they're pixels that take a lot of time and effort to acquire. Because of this, people take things fairly seriously.

      In this case - everyone knows that CCP has some issues. There are known problems that people just work around - like Jita on the weekends. You know

  • by Rakshasa Taisab ( 244699 ) on Tuesday January 05, 2010 @01:22AM (#30651812) Homepage

    It's well known and not even contested that the forces bridging in to the system black-screened and never got to fight.

    However, they got what they deserved. The node in question was not reinforced due to the unexpected nature of the fight (as in; the notification system was not used to put the system on a dedicated server). And jumping into large fights was well know to be bugged since the expansion and the Fleet Commander was made aware by an alliance member that the specific way in which they were going to enter the fight would trigger the bug.

    They ignored all those warnings and decided to go ahead. Sources claim the intent was to crash the node and get a more even fight once it got up, multiple accounts even got banned for spamming local chat. Funny thing is the bug seems to be in the simultaneous transfer of 100+ ships into an overloaded system, and doesn't affect people warping around within a system once they are there. This being the worst possible situation for the attempted rescue of the system.

    • by Anonymous Coward on Tuesday January 05, 2010 @01:41AM (#30651904)

      For those who don't know, there were already 750-800 players in the system when the defending forces decided to jump into the system. It was a stupid move on the part of the commanders, and they deserved to be shot down like they were.

      EvE has many thousands of systems, and many have very few players in them a large portion of the time. CCP requests that when large alliances are going to have a large fight that they notify them so they can put the system in question on a more powerful server to support the large number of players.

      Goonswarm, PL, and SOT knew that the system in question was going to be attacked and failed to inform CCP. After they lost the race to get players in the system first, they decided to attemp to crash the node by spaming the local chat channel and jumping everyone in at the same time, and then beat IT alliance back into the system when the server came back up, but they failed, and lost a record dollar value of ships for one fight. Before that fight there had only been about 15-20 titans killed across the whole game, and they lost 4 in one fight. Pretty epic.

      • Seems the next project for the EVE programmers is dynamic reallocation of their solar systems based on number of people in local to a smaller set of high-performance systems.

        Either that -- or the server whose load goes up (people in local climbing over treshhold) should start offloading lower priority solar systems it also hosts to other servers to reduce its CPU/Memory usage.

        • by TopSpin ( 753 ) *

          Maybe. Dynamically moving systems to faster CPUs will raise the ceiling a little. The game is growing; the big alliances have 4000+ players. Eventually they'll break the fastest CPUs CCP can get their hands on.

          The servers use "cooperative multithreading" (their term, not mine) which means it can't be distributed across cores because the system isn't thread safe. Read about it here [slideshare.net].

          EVE just doesn't scale. Microthreads, green threads, whatever you want to call them, are elegant and efficient while your p

    • Re: (Score:2, Informative)

      by piggydoggy ( 804252 )
      Sadly the node couldn't be reinforced in time, as CCP's policy requires it to be done during the daily downtime. Yet thanks to a screwup by the original owners of the system, the attackers (in this battle) had only 12 hours to make their move and attack the system - not enough to wait for the next downtime and node reinforcement.
  • The scene where Kirk is facing off against Spock's unbeatable scenario and shuts down the servers temporarily, disabling the shields all of Spock's ships, before blowing them up.

  • "annihilated amidst controversy"? When has there ever been any significant battle in Eve that didn't feature people blaming server issues for their loss? (often correctly, I might add)

  • EVE Online. (Score:4, Informative)

    by MindlessAutomata ( 1282944 ) on Tuesday January 05, 2010 @01:30AM (#30651868)

    I'm not trolling, but I fail to see the point of EVE for several reasons. I used to play EVE myself for a few months but quit...

    One, why play a game that takes you at least a year to be able to do anything fun and useful? That's not a game at all, that's a job.

    Two, CCP has shown themselves in the past to be shady and unreliable, having developers specifically favor certain alliances and otherwise abuse their powers for their own in-game corporations.

    Three, the amount of bugs and inability to cope for server stress for large battles (which is the meat and potatoes of this game--large space wars!) has apparently been evident for quite some time now.

    I understand that EVE online fills a niche few other games do, and EVE is probably the only one that even attempts what it does, but, IMO, that in no way means the CCP has displayed what I would consider a necessary amount of competence or good game design to make me want to play it. I mean, if Age of Conan (no, EVE is nowhere near the mess that game was at) was the only MMO out there I still wouldn't play it even though I like MMOs.

    It's pretty poor form when CCP will claim that subscribers need to account for their own ineptitude when playing their game and not take responsibility for their own, and not even fire the developers that gave unfair advantages to their own corporations way back when. And I hear the game masters are incompetent jerks, too...

    • It's a game for people that want to treat a game as a job, and who think PvP is the only real fun out there. Ie, hyper-competitive people that tend to be scary in real life too.
    • Re:EVE Online. (Score:5, Informative)

      by DMUTPeregrine ( 612791 ) on Tuesday January 05, 2010 @01:51AM (#30651974) Journal
      1) It takes about a month to fly an interceptor. I'd far rather have a newbie in an interceptor than a battleship, they'll be worthless learning in the battleship. This is just plain wrong.
      2) This has vastly improved with the creation of the internal affairs department. The problem with CCP is more incompetence (mostly on the part of the low-level GMs) than outright malice.
      3) The Reinforced node system helps, but is too limited. This is honestly the biggest issue with the game. The servers need to be able to support the player base.
      I play EVE because I enjoy small scale combat with meaningful risks. If I wanted to have epic battles with thousands of ships I'd probably be disappointed in it, but for 10-20 man roaming gangs it's very fun.
      • Re:EVE Online. (Score:5, Interesting)

        by cbhacking ( 979169 ) <been_out_cruisin ... nospAM.yahoo.com> on Tuesday January 05, 2010 @02:39AM (#30652234) Homepage Journal

        Mod parent up!

        You can have enough skill points to be useful in 0.0 (the unsecured space where players make all the rules and large alliances carve out empires) before the end the free trial. Sure, you won't be able to kill the most valuable NPCs or take on almost anybody in a solo fight, but you can make more than enough money to support buying the ships and gear that your skills allow you to use, and you can certainly be useful in a roaming gang or defensive camp. Heck, you might even get lucky and find some idiot with a hauler full of valuables and nobody escorting him (happened to me once) in which case you really only need a warp disrupter (cheap and easy to train for) and enough firepower to overcome the hauler's shield recharge rate (which you could get by your second day of playing the game).

        EVE and CCP may never completely live down poor decisions on the part of several employees, but the game itself goes on and for most people such events are scarcely newsworthy for a week. While we'd certainly prefer if such things had never happened, they're old news - almost irrelevant by now - and the CCP has taken some fairly solid steps to prevent such things from happening again.

        From the sound of it, this fight was executed wrong in almost every possible way, perhaps most importantly in that CCP wasn't notified ahead of time so they could put the system on high-end dedicated hardware. Consider also that having hundreds of people in the system used to be enough, by itself, to cause atrocious lag (even if they weren't fighting one another), a problem which is very rare today. Now, while fights with nearly 1000 player/side might still be a bit more than the game can handle, a few hundred per side is commonplace and a thousand total is well within the capabilities of the "reinforced" (with dedicated servers) nodes.

        • I never said you can't be useful in low level ships, you can--but your job basically is slowing other ships down. Not fun.

          • why play a game that takes you at least a year to be able to do anything fun and useful?

            I never said you can't be useful in low level ships

            You know we can all read what you just wrote, right?

    • Re: (Score:2, Informative)

      by techhead79 ( 1517299 )

      A year to do anything useful? That's what you got from playing for a few months?

      I don't really see any difference between that and say WoW where what everyone tells you the first goal is to get to level 80. How long does that take you assuming you don't have a pal leveling you through everything or you were not so devoted that you went out and found a leveling guide and followed it to the letter. All games like this take HUGE amounts of time. I don't see how eve is any different.

      The major problem with eve

    • Re:EVE Online. (Score:4, Interesting)

      by jeff4747 ( 256583 ) on Tuesday January 05, 2010 @02:26AM (#30652150)

      Two, CCP has shown themselves in the past to be shady and unreliable, having developers specifically favor certain alliances and otherwise abuse their powers for their own in-game corporations.

      Three, the amount of bugs and inability to cope for server stress for large battles (which is the meat and potatoes of this game--large space wars!) has apparently been evident for quite some time now.

      IMO, these two issues are caused by CCP's location. There's not a vast number of high-quality programming talent in Iceland. (Simply because there isn't a vast number of people). CCP has had a "Senior Programmer" open position on their web site for a couple of years now.

      Frankly, the reason I left EvE is that the quality of the already mediocre code was heading downhill rapidly. My personal tipping point was the bug where in-game browser bookmarks were not properly imported by their new in-game browser. If the Devs can't even do that right, then how can anyone expect much from the rest of their code?

    • Re:EVE Online. (Score:4, Informative)

      by Korin43 ( 881732 ) on Tuesday January 05, 2010 @02:26AM (#30652158) Homepage
      I don't play anymore, but I think you're wrong about your first point. One of the reasons the Goonswarm is so powerful is that they recruit noobs in frigates and cruisers and then swarm the enemy. It's not like other games where a level 70 player could stand around being hit by a lower level character all day. You can destroy a titan using frigates (and the titan likely wouldn't even be able to fight back, it's only options would be to call for help or run away). If you think you need to have a big ship to do anything fun or "be useful", you're missing the point.
      • I said FUN and USEFUL, not fun and/or useful. Big difference--swarming the enemies and slowing them down is not fun, it's mindless and tedious.

  • by toxygen01 ( 901511 ) on Tuesday January 05, 2010 @01:42AM (#30651912) Journal
    • Re: (Score:3, Funny)

      by Korbeau ( 913903 )

      I have no clue about how this game work, but judging by the video the big vessels clearly should have gone into Ludicrous Speed mode!

    • by routerl ( 976394 ) on Tuesday January 05, 2010 @03:07AM (#30652356)
      That really looks terribly boring. After decades of big budget sci-fi movies, not to mention epic space battle video games like Homeworld, this is the best space combat system that EVE can offer? There didn't seem to be any maneuvering involved at all... might as well be a text based game.
      • Re: (Score:2, Informative)

        by Anonymous Coward

        That really looks terribly boring. After decades of big budget sci-fi movies, not to mention epic space battle video games like Homeworld, this is the best space combat system that EVE can offer? There didn't seem to be any maneuvering involved at all... might as well be a text based game.

        That's because you don't see any interceptors or for that matter any other small craft flying around. That might be a matter of scale or maybe their display is turned of to prevent the death of the graphics card. What you SEE is a big lump of imperial star destroyer sized ships and some death star sized ships which shouldnt move fast.

      • Re: (Score:3, Insightful)

        by Opportunist ( 166417 )

        It was basically a one sided battle, shooting fish in a barrel because the other side was caught in lagland (server transition isn't so great in EvE when a few hundred ships are already in the system). So I guess if you're expecting to see a "battle", you have to end up disappointed. It's a bit like shooting sitting ducks that had their wings cut off.

      • Look again.

        There are what? 300+ huge ships all clustered together in a box that appears to be no larger than maybe 10 ships on each side. It's absolutely cramped. Where should they go? If you're in the middle, you're fucked, because if you start moving about, you've just turned yourself and the ships around you into kinetic missiles.

        Those on the outer edges of the box could move around, sure. They can back away. But if they have to turn around to face their engines towards the centre mass and their weapons

    • Re: (Score:3, Interesting)

      by seifried ( 12921 )
      I don't get it. Everyone just parks their ship and slugs it out? No maneuvering? Boring. Gratuitous Space Battles has way better game play.
      • Re: (Score:3, Informative)

        by Opportunist ( 166417 )

        You're looking at effing HUGE ships. There isn't much movement sensibly possible for them, they are slow and turn even slower. Battles in EvE are at this scale less a matter of maneuvering and moving yourself in position (ships that depend on that instead of huge hulls exist, too, and they serve a vital role, too). Battles of this size are highly influenced by actions before the battle even starts, they're more a matter of good logistics and efficient group management before, as well as good coordination of

        • You can try and convince yourself about all you want, but "logistics" means joining the biggest gang, then getting into the system first and chewing up the attackers piecemeal. If that's your idae of "efficient group management", then sure, it's undeniably effective.

          Funny that you should mention screensavers; I've lost count of the number of people who have described EvE as a very pretty screensaver.

          • Personally I see it as Excel with better graphics. But then again, I'm in resources...

            Logistics is much more than that. Like, making sure you actually have what you need when you need it. Just having 2000 people in your alliance means jack if they have no resources, which means no place to produce, which means no ships. Likewise, it means jack if they're scattered all over the galaxy. Logistics is about having what you need where you need it when you need it.

            This may not be a concern to alliances that shrug

      • Re: (Score:3, Insightful)

        by thoth ( 7907 )

        As others have noted, you're watching the equivalent of aircraft carriers fighting each other - those don't dodge or pivot much.

        More exciting action is much scale, 5 on 5 or fewer. In those type fights, one person will try to slow a ship (web stasis field), prevent it from escaping (warp disruptor), make it more visible electonically (target painting), scramble it (electronic scrambling), drain its energy (nosferatu modules), etc. Though set in a sci-fi game, smaller scale battles play out reasonably real

      • Re: (Score:3, Informative)

        by zerocool^ ( 112121 )

        This gives you an idea of the size of the ships involved:
        http://www.gossipgamers.com/images/eve1.jpg [gossipgamers.com]

        Anyway, yeah, Titans are huge, and the rest of the ships there are dreadnoughts mostly (they're also huge). Check the chart - in my opinion, most of the fun combat is in cruiser sized ships. Find the Caracal or the Rupture or the Vexor to see a size comparison. There's lots of maneuvering with those kind of ships.

        Something like http://www.youtube.com/watch?v=Aeyt-T_U2Vg [youtube.com] gives you a better idea, but even th

    • Here's a screenshot [ggpht.com] from the losing side!
    • Is that a laser Floyd show or a video game?
  • by Antiocheian ( 859870 ) on Tuesday January 05, 2010 @02:30AM (#30652174) Journal

    Gamers with a tighter schedule (work, studies, family etc) or a lagging connection to online servers should really consider an offline alternative that goes with their own pace and allows time speed adjustment. Without time speed adjustment (which is by definition incompatible to large online games) space games can be extremely time consuming.

    X3 Reunion + Xtended mod (I didn't like TC very much) is a good alternative but I'd be willing to know more.

  • Props to CCP for contributing to the development of "stackless python", but maybe they should write their server code in a different language.

  • The allegations that IT (the victorious side in the battle) was the "successor alliance" of BoB (to inform the ones not caring too much about EvE: The alliance that allegedly had undue and "illegal" help from CCP insiders) and that it's the whole BoB inside job deal again are already starting.

    I love EvE. It's one of the few games where just watching the metagame, rumors and drama around it can be at least as entertaining as playing the game itself.

  • OP is a bit lame as it fails to even mention Pandemic Legion who
    lost the 4 titans.

    Pandemic Legion loses 4 titans [kugutsumen.com]
    ( contains teamspeak recordings of the attack, screenshots, chatlogs, etc...)

    Bobbechk's comic strip about the event [kugutsumen.com]

    EVE-O Uncensored Daily Political Updates [kugutsumen.com] (reliable source of information about EVE Online politics, updated daily.)

  • Clarification (Score:4, Informative)

    by Zos23 ( 1614073 ) on Tuesday January 05, 2010 @09:01AM (#30654170)
    A lot of what was said here is incorrect. EVE usually allows for very nice fleet battles with small amounts of lag. However about a month ago a new expansion was introduced that includes a nasty bug which makes it extremely hard for people to load the grid that already contains many other players. There were several battles in the past month where one side was completely annihilated because of that bug. Everyone involved in that conflict was already aware of that. IT alliance had a strong presence in system for the whole day, preparing for possible battle. Their enemies decided to show up when it was almost over while boasting about crashing the node. Sov was neutral and both parties had the same starting position. IT was bringing in forces during the whole day, the other side did nothing about it and gambled it all on one moment and lost.

How many QA engineers does it take to screw in a lightbulb? 3: 1 to screw it in and 2 to say "I told you so" when it doesn't work.

Working...