EVE-Online Patch Makes XP Unbootable

Nobo writes "CCP's latest major patch to the EVE-Online client, Trinity, comes with an optional DX9-enhanced graphics patch that dramatically improves the visual quality of the in-game graphics through remade models, textures, and HDR. It also has an unfortunate bug: the incredibly stupid choice of boot.ini as a game configuration file, coupled with an errant extra backslash in the installer configuration. The result is that anyone who installs the enhanced graphics patch overwrites the windows XP c:\boot.ini file with the EVE client configuration file, bricking the machine on the next boot. Discussion in a couple of forums threads is becoming understandably heated."

Insanely sloppy... but not without precedent

[RogueyWon (735973)]

Wow... if this story isn't a wild exaggeration, then this is about as unfortunate as a game-bug can possibly get. Of course, a reasonably savvy user could probably have an affected system working again fairly quickly without any data-loss, but my own experience suggests that such users will be in the minority.

The only gaming-related parallel I can think of relates to the uninstall programme bug for the 2001 version of Pool of Radiance. In that instance, attempting to uninstall the game (something many users would do not long after installing it, given the tedious and half-baked nature of the game) had a good chance of wiping the user's hard disk. I actually deliberately triggered this bug for fun myself when I decided it was time to wipe my old machine after I bought a new system. If anybody can think of any other examples on this kind of scale, please do share them.

I wonder if this is going to cause any unpleasant and potentially expensive legal repercussions for CCP, from users who have lost data while trying to fix the issue?

Re:Insanely sloppy... but not without precedent

[iCEBaLM (34905)]

At one point trying to uninstall Final Fantasy XI Online would remove hal32.dll.

Re:Insanely sloppy... but not without precedent

[gEvil (beta) (945888)]

At one point trying to uninstall Final Fantasy XI Online would remove hal32.dll.

That wouldn't be a smart thing to do, now would it, Dave?

Re:Insanely sloppy... but not without precedent

[sm62704 (957197)]

Don't these dumbasses actually test this shit before they shovel it out to you? It seems that a patch that would brick XP would be a bug that the first goddamned time it was tried would be discovered. I'm going to have to google to see what company makes Final Fantasy so I can be sure never to buy a game from them!

The love of money is the root of all bad software.

-mcgrew

Re:Insanely sloppy... but not without precedent

[TheThiefMaster (992038)]

Uninstallers and patches are rarely tested fully. For patches, normally problems stem from the company having only ever tested the clean game at the latest build, or having only tested patching from a clean install of the original retail copy.

Also, this EVE patch wouldn't "brick" an XP SP2 machine that had Windows installed to the primary partition of the primary drive (i.e. most pcs), because Windows XP SP2 will automatically try to boot that if it fails to find boot.ini. Assuming they did test the patch, this would explain why they didn't notice.

Re:Insanely sloppy... but not without precedent

[malkavian (9512)]

Because the file existed before, and after the patch, and because (as the GP poster mentioned) their clean test machine would continue to boot cleanly after the install.
I can understand this one slipping through the cracks in coding (having done coding for years, and knowing that something like a game doesn't get full formal spec treatment). It's still a big ouch, and a real hit on the reputation for the company, but it's one of those honest to god accidents of oversight.

If it happened to me, I'd be mightily peeved, and rightly so. As the company will likely be frantically running round trying to sort it out, and being both scared and embarrassed.. And again, rightly so.

The measure of the company now is in how well they manage the screwup, and how well they look after the people affected. Accidents can happen to anyone. Not everyone can manage to do something good about a disaster.

Re:Insanely sloppy... but not without precedent

[TheThiefMaster (992038)]

It's not, because the summary is wrong.

The patch actually deleted the system boot.ini, it doesn't over-write it or replace it with a game config file.

I don't know where that "fact" came from.

Trust me, I was one of the people who had their boot.ini deleted by the patch, followed by (on next boot) my machine displaying some warning about boot.ini being missing, and then proceeding to boot anyway.

Re:

[IndustrialComplex (975015)]

I wonder if this is going to cause any unpleasant and potentially expensive legal repercussions for CCP, from users who have lost data while trying to fix the issue?


At the very least, it will give us a better indication of just how binding those EULAs are.

With respect to the bug, I'm an ex-tech. I've spent so long away from tinkering with my OS that it would probably take me a good long time to realize just what was wrong. I could probably repair the machine once I did find out that it was a boot.ini iss

Re:

[encoderer (1060616)]

The problem is that your post is gibberish to most users.

For most users their choice is binary:

0. Call the family IT guy (you know, one of us..) and waste our time (as if we don't sit in front of a PC enough..)
1. Call Geeksquad or a similar ripoff-artist and pay $100+ to have them wipe the disk and re-install windows, after stealing all your porn and music

Re:Insanely sloppy... but not without precedent

[by Anonymous Coward]

The deletion of the Boot.ini file will not cause any data loss. If you format your system to fix the issue then you will lose data. Anyone with the Windows XP CD can boot off of it and repair the OS. It is a simple procedure for the tech savvy folks and for those that are not tech savvy, most of them have friends that are.

This issue is going to leave CCP with a lot of egg on their face but realistically extended downtime would have been worse since the player base would have been screaming a 100x louder. This issue will peak higher in the media since it is a highly unusual problem but will die quicker then if the servers were down for 2-5 days.

The concern that I have is how did this get past the QA testers at CCP and into a production build?

Re:(catchy subject)

[Krinsath (1048838)]

More likely cause: The patch was tested but the patcher was not.

Don't forget that this is an issue with the the *patcher* that was not present in the full premium install from scratch, only the upgrade (which is probably the route most people would've taken, in fairness). It basically boils down to a simple typo in one version of the installer and rebooting to test the installer might not be part of their QA tests for the patcher.

Really what they should catch flak for is not a bad typo, but as the summary points out having a game file with the same name as a critical OS file. Boot.ini isn't a new thing, in fact it is on its way out with Vista, so there's really no excuse to claim you didn't know that Windows had such a file. It's been there since 1995 or so.

Oblig. Penny-Arcade

[Drogo007 (923906)]

From way back in 1999 with good ol' Myth II;

http://www.penny-arcade.com/comic/1999/01/06 [penny-arcade.com]

I remember when we used to use this strip in our training materials for new Testers to impress upon them how badly they did NOT want to have a comic like this made about a bug THEY missed.

Re:Insanely sloppy... but not without precedent

[Fozzyuw (950608)]

I still can't figure out how this missed even basic testing where CCP should have caught this bug pretty easily.

The testers would have caught it but their computers didn't start when they tried to turn them on the next day so they could never identify it. =P

Ppffftt!

[morgan_greywolf (835522)]

Isn't this something should have been found in, oh, I dunno....beta testing?

Re:Ppffftt!

[Phisbut (761268)]

Isn't this something should have been found in, oh, I dunno....beta testing?

Oh, but it was found, by several beta testers. However, since none of those beta testers had a functioning computer after the test, they were all unable to send a bug report. Not having received any bug reports, the developers simply assumed that there were no bugs.

Re:Ppffftt!

[kv9 (697238)]

Consumers==Unpaid Beta Testers

paying beta testers

How is this possible?

[Jennifer York (1021509)]

Someone in their QA department needs to be fired. This type of mistake is simply unacceptable, and truly very difficult to believe.

What sort of test plan fails to catch BRICKING THE PC?

Re:How is this possible?

[vranash (594439)]

Obviously one with a really high uptime for Windows :)

Re:How is this possible?

[sayfawa (1099071)]

Say John, there's a funny thing with our new patch; after the dialogue telling the user that the install was successful and that they should reboot the machine, the machine doesn't actually reboot, it just shuts off and then hangs. What should we do?

Don't tell them to reboot the machine. Problem solved.

Re:

[faloi (738831)]

The test plan where you throw it on your Vista box and test it, and it works fine (Vista doesn't use boot.ini), then you test your other OS clients. After all, it's just the installer, what could go wrong...*cough*

Re:How is this possible?

[Jugalator (259273)]

Bricking. That word will have became annoying to me by the end of 2008.

Re:How is this possible?

[MBGMorden (803437)]

I think it became annoying to me at the end of November. What's bad is it's usually not the geeks who fall in love with buzzwords.

Re:How is this possible?

[illumin8 (148082)]

BRICKING THE PC?

WTF is with you people? Ever since the Apple made iPhones "bricks", this erroneous use of the term has seeped into our technical vocabulary. People, it's not a brick if it's still usable. When a piece of electronics is really bricked, that means that the ROM is in such an unrecoverable state, that it can't even be flashed with a new working ROM, and needs to be either thrown away, or sent to a factory for repair.

Now, the term bricking is being applied to any piece of electronics or computer equipment that won't boot an OS.

It's not bricked if you can just reinstall or repair Windows and have it work again. It's bricked if you flash a bad ROM BIOS image and now you can't even turn the thing on.

Re:How is this possible?

[jmoriarty (179788)]

WTF is with you people? Ever since the Apple made iPhones "bricks", this erroneous use of the term has seeped into our technical vocabulary.

Sheesh... way to brick the discussion...

Re:How is this possible?

[garbletext (669861)]

I have mod points to give you but you're already at +5. Thank you for giving voice to my frustration over this usage. Imprecise language helps no one. A device is called a brick because it is no more useful than one. If you can fix it, it's just 'broken.'

Re:How is this possible?

[Bill, Shooter of Bul (629286)]

No, thats just stupid. A computer with a missing boot.ini is much more useful than a brick. Case in point, you can buy a bare bones computer with no OS for ~ $400. More than your average masonry brick. If people do sell it a state similar to its "bricked state" for more significantly more than a brick, then I would say its not bricked. Because, obviously, its more useful than a brick.

Re:How is this possible?

[garbletext (669861)]

No. Bricking doesn't refer to the user's ability to operate a given technology, otherwise my grandma could call her new microwave "bricked," which isn't the case. Just because you are too incompetent to use something doesn't mean it isn't useful. A brick is a device that is irreparably broken.

Re:How is this possible?

[rfunches (800928)]

No, the term "brick" does not change based on your technical experience and "considering" something to be bricked does not make the use of the term correct. Joe Average may refer to his hard drive as "memory" but his use of the term is still inaccurate. If the flash chip on an iPhone is FUBAR'd to the state where you can't even reflash it by any means, it's bricked, whether it's in Joe Average's hands, Steve Job's hands, or Sally Tech's hands. Anything less than rendering a piece of hardware completely inoperable (hardware with the usefulness of a physical brick) is *not* bricked. Now, if the boot.ini removal rendered a hard drive inoperable...

They both made errors.

[d3m0nCr4t (869332)]

I suppose both the producers of Eve Online and MS are to blame here. Eve Online for naming a configuration file the same as a Windows system file. And of course MS, for letting any application overwrite such an important system file.

Re:They both made errors.

[W2k (540424)]

Likely the users were running the game as administrators, and an administrator would have the necessary rights to overwrite any file on the disk. I don't see how this could be blamed on Microsoft. On Vista you'd get a UAC prompt for trying to write to C:\, but Vista doesn't use a BOOT.INI anyway, so no risk of breaking the system.

Re:They both made errors.

[Goobermunch (771199)]

Sad that so many games require Administrator access to run.

--AC

Re:They both made errors.

[00lmz (733976)]

It certainly is sad that some apps and games need admin privs to run, but this is an installation bug. Of course people are going to install programs as administrator...

Re:They both made errors.

[secPM_MS (1081961)]

There is nothing that Microsoft could have done to prevent this. Installation of applications to the machine requires administrator privledges, as does installation of drivers. On Vista, there will be a UAC prompt when you start installation and uninstallation, but the process will then run with the full administrator token. Admin's can do what they want on the box. On a *nix system such an installation / uninstallation error would typically nail the system as well unless it was run in a rather full jail, and I am uncertain that jailing the game would have adequately dealt with a process that might install new video drivers. Certainly, most users would have been slammed in either environment.

Microsoft is criticized for its slow release of patches and software. One of the major issues slowing down release is the exhaustive testing passes that software must go through, and they still occasionally miss something. The diversity of configurations in the field is astonishing. This is an issue Apple does not face, as they support an OS for ~ 2 .releases, say 3 years -- and they make all the HW, which limits the diversity. Microsoft supports their stuff for 7 to 10 years (the 9X and ME series were a bit less than this).

Bricking?

[interactive_civilian (205158)]

Why does the summary say "bricking the machine"? Does the machine become a doorstop that cannot be fixed? Can you not (and this might even be more complicated than necessary, but as a rather inexperienced Windows user, this came to mind first) use a Linux Live CD to boot and edit the necessary files? I DNRTFA, but if it is just an errant backslash, it should be a piece of cake to fix.

Hardly "bricking" IMHO.

Re:

[IceCreamGuy (904648)]

You are absolutely correct, you can use a Linux live CD, a BartPE disc, the Windows install disc, whatever you have that can access an NTFS partition. It's a pretty easy procedure, the equivalent of rewriting a grub config file, just need to know the %windir% folder and installed partition. Brick is definitely not an accurate description.

Re:Bricking?

[geminidomino (614729)]

As I mentioned in my previous post:

People with one machine and w/o a Linux live CD (probably 90% of windows users) would have a bricked machine barring any outside assistance.

No, they wouldn't. The term "bricked" has very specific connotations. Specifically, that it is not repairable without professional intervention which will probably cost more than the unit itself, thus turning it into a "very expensive brick."

A crashed OS is not a bricking, unless that OS is on firmware or something. If popping in a CD can fix your computer, whether or not you are too stupid to do it yourself, then it's not bricked.

Re:Bricking?

[gazbo (517111)]

Man, that's serious then. One would have thought that MS would make the Windows CD bootable so that users could gain access to some form of "recovery console".

Re:

[MMC Monster (602931)]

Well, yes they do that, but not in any useful way. (At least on WinXP.)

The recovery console is shell-based. Which sucks, since MSWindows likes to keep most of it's recovery tools GUI-based. (Can you even edit the registry in a console?) Or you can allow the installation disc to "fix" the installation, in which case you don't have any control of what it does.

Partially correct story

[NATIK (836405)]

Everything the newsstory says is correct, but the issue have been fixed and anyone updating now wont get hit by it.

It is still a momumental fuckup though and the one responsible needs to be kicked in the balls for that kind of stupidity.

It's not bricked!

[wiredog (43288)]

Dammit! When did "bricking" expand it's meaning from "unbootable under any conditions due to firmware (such as the BIOS) being improperly overwritten" to "Oops, have to pull out the rescue CD"?

Re:It's not bricked!

[halcyon1234 (834388)]

I think we need to start subclassing the term. It's bricked, but it's recoverable. So it's just a mild inconvenience. A nerf brick? Loose grout?

Apologies

[Sockatume (732728)]

In Parlimentary Republican Iceland, game breaks Windows!

Not sure this is a QA problem...

[E. Edward Grey (815075)]

Things like this can easily happen when your patch doesn't have any CHANGE CONTROL. Imagine this - the patch is ready to go, everyone agrees on it, and then a small group of developers (or maybe even a single developer) decides to make a modification...and implements it badly. It doesn't even go through QA because QA isn't invoked ("oh, that would just delay the release, I'm sure I have it right anyway"). And now you have this.

I know it drives us crazy, I know not every organization implements change control that's sane and logical. But there's a reason it exists!

Eve's boot.ini

[splutty (43475)]

The boot.ini for Eve itself contains information about whether you have the "Classic" version or not. The patch that was released for the Classic version did not contain this problem.

The patch released for the "Premium" version does contain this installer error. The change made to the boot.ini is the line that contains this definition, and is changed from Classic to Premium.

It's a very logical problem, easy to fix if you know it, but also incredibly stupid...

Alarmist

[Sobrique (543255)]

If you don't install your games to C: you're fine.
If you've got a 'basic' OS install, e.g. C:\WINDOWS and one partition, you're fine - the boostrap loader guesses, flashes up an error, and boots anyway.
It's a bit of a fubar, but hardly the next apocalypse.

Re:Bricking? BS! More FUD!

[beheaderaswp (549877)]

You and I are probably both competent technical people. For my part, I'm an IT director and have done this type of work for 22 years.

Let us assume the two of us, you and I, know more about the Windows registry, bash shell, or using gcc that 98% of the geeks out there. Just for argument's sake.

However, there's a 95% chance that any EVE online player will have the following qualities:

1. Own only one computer.

2. Not be technical.

3. Not read the forums where the information is posted.

4. Be unable to digest and properly utilize the fix information.

So let us re-asses:

It took us, you and I, about 15 seconds to re-write that boot.ini file and *poof* no problem.

That's 5% of the EVE userbase. Add another 20% of the userbase that figures out how to solve the problem. 25% of the people have the fix.

The rest of those poor schlubs are driving to Best Buy to have some incompetent charge them $100 (or whatever)- and that is NOT FUD!!

That my friend is a screwup of massive scope, with huge consequences, because for people who are not geeks- that computer is a "brick".

Re:

[Tim C (15259)]

Plus I expect they have permissions in place to prevent the overwrite.

If they're not installing stuff as administrator, they should be. If they've modified their machine significantly from what the average gamer would have, they shouldn't have (by which I mean going in and denying even administrator access to system files, for example).

Besides which, another poster claims that the EVE boot.ini file contains specific information about which version of the game you have, and that it's only installed by the

Re:

[Jason Levine (196982)]

Except that:

1 - Many systems don't come with XP CDs anymore. They come with "restoration partitions" that revert the entire system to a default factory state and might incur data loss.

2 - I'd bet that most users wouldn't know how to use their XP CD or restoration partition if they needed to.

So, yes, messing up the OS this bad would be "bricking" the computer for these users. Sure the fix is simple to you and me, but it's horrendously technical to them. This doesn't even get into the fact that these peopl

Re:

[Malc (1751)]

Get off the let's blame Microsoft bandwagon. This was the installer not the final app. At very least it would have required admin rights to write under "Program Files", because MSFT does enforce security.

"about 80% of the users run with admin privileges on XP, because most apps simply don't work as standard user"

Wrong. I've been logging on to XP as a limited user for years. Most apps work. Some broken apps can be made to work by fiddling with NTFS and registry permissions (hardly ideal, but workable).

Not a brick, dammit!

[CoreDump01 (558675)]

A bricked device either to be sent in to the vendor for repairs, or ,as an alternative, can only be revived via special debugging hardware by people with god-like skills in a certain areas.

A blown OS is not, and never ever will be a brick. Get your terminology straight for once. Wikipedia explains rather nicely the nature of real "brick".