EVE-Online Patch Makes XP Unbootable

Nobo writes "CCP's latest major patch to the EVE-Online client, Trinity, comes with an optional DX9-enhanced graphics patch that dramatically improves the visual quality of the in-game graphics through remade models, textures, and HDR. It also has an unfortunate bug: the incredibly stupid choice of boot.ini as a game configuration file, coupled with an errant extra backslash in the installer configuration. The result is that anyone who installs the enhanced graphics patch overwrites the windows XP c:\boot.ini file with the EVE client configuration file, bricking the machine on the next boot. Discussion in a couple of forums threads is becoming understandably heated."

Insanely sloppy... but not without precedent

Wow... if this story isn't a wild exaggeration, then this is about as unfortunate as a game-bug can possibly get. Of course, a reasonably savvy user could probably have an affected system working again fairly quickly without any data-loss, but my own experience suggests that such users will be in the minority.

The only gaming-related parallel I can think of relates to the uninstall programme bug for the 2001 version of Pool of Radiance. In that instance, attempting to uninstall the game (something many users would do not long after installing it, given the tedious and half-baked nature of the game) had a good chance of wiping the user's hard disk. I actually deliberately triggered this bug for fun myself when I decided it was time to wipe my old machine after I bought a new system. If anybody can think of any other examples on this kind of scale, please do share them.

I wonder if this is going to cause any unpleasant and potentially expensive legal repercussions for CCP, from users who have lost data while trying to fix the issue?

Re:Insanely sloppy... but not without precedent

At one point trying to uninstall Final Fantasy XI Online would remove hal32.dll.

Re:Insanely sloppy... but not without precedent

At one point trying to uninstall Final Fantasy XI Online would remove hal32.dll.

That wouldn't be a smart thing to do, now would it, Dave?

Re:Insanely sloppy... but not without precedent

Don't these dumbasses actually test this shit before they shovel it out to you? It seems that a patch that would brick XP would be a bug that the first goddamned time it was tried would be discovered. I'm going to have to google to see what company makes Final Fantasy so I can be sure never to buy a game from them!

The love of money is the root of all bad software.

-mcgrew

Re:Insanely sloppy... but not without precedent

Uninstallers and patches are rarely tested fully. For patches, normally problems stem from the company having only ever tested the clean game at the latest build, or having only tested patching from a clean install of the original retail copy.

Also, this EVE patch wouldn't "brick" an XP SP2 machine that had Windows installed to the primary partition of the primary drive (i.e. most pcs), because Windows XP SP2 will automatically try to boot that if it fails to find boot.ini. Assuming they did test the patch, this would explain why they didn't notice.

Re:Insanely sloppy... but not without precedent

Because the file existed before, and after the patch, and because (as the GP poster mentioned) their clean test machine would continue to boot cleanly after the install.
I can understand this one slipping through the cracks in coding (having done coding for years, and knowing that something like a game doesn't get full formal spec treatment). It's still a big ouch, and a real hit on the reputation for the company, but it's one of those honest to god accidents of oversight.

If it happened to me, I'd be mightily peeved, and rightly so. As the company will likely be frantically running round trying to sort it out, and being both scared and embarrassed.. And again, rightly so.

The measure of the company now is in how well they manage the screwup, and how well they look after the people affected. Accidents can happen to anyone. Not everyone can manage to do something good about a disaster.

Re:Insanely sloppy... but not without precedent

It's not, because the summary is wrong.

The patch actually deleted the system boot.ini, it doesn't over-write it or replace it with a game config file.

I don't know where that "fact" came from.

Trust me, I was one of the people who had their boot.ini deleted by the patch, followed by (on next boot) my machine displaying some warning about boot.ini being missing, and then proceeding to boot anyway.

Re:Insanely sloppy... but not without precedent

The deletion of the Boot.ini file will not cause any data loss. If you format your system to fix the issue then you will lose data. Anyone with the Windows XP CD can boot off of it and repair the OS. It is a simple procedure for the tech savvy folks and for those that are not tech savvy, most of them have friends that are.

This issue is going to leave CCP with a lot of egg on their face but realistically extended downtime would have been worse since the player base would have been screaming a 100x louder. This issue will peak higher in the media since it is a highly unusual problem but will die quicker then if the servers were down for 2-5 days.

The concern that I have is how did this get past the QA testers at CCP and into a production build?

Re:(catchy subject)

More likely cause: The patch was tested but the patcher was not.

Don't forget that this is an issue with the the *patcher* that was not present in the full premium install from scratch, only the upgrade (which is probably the route most people would've taken, in fairness). It basically boils down to a simple typo in one version of the installer and rebooting to test the installer might not be part of their QA tests for the patcher.

Really what they should catch flak for is not a bad typo, but as the summary points out having a game file with the same name as a critical OS file. Boot.ini isn't a new thing, in fact it is on its way out with Vista, so there's really no excuse to claim you didn't know that Windows had such a file. It's been there since 1995 or so.

Re:Insanely sloppy... but not without precedent

I still can't figure out how this missed even basic testing where CCP should have caught this bug pretty easily.

The testers would have caught it but their computers didn't start when they tried to turn them on the next day so they could never identify it. =P

Ppffftt!

Isn't this something should have been found in, oh, I dunno....beta testing?

Re:Ppffftt!

Isn't this something should have been found in, oh, I dunno....beta testing?

Oh, but it was found, by several beta testers. However, since none of those beta testers had a functioning computer after the test, they were all unable to send a bug report. Not having received any bug reports, the developers simply assumed that there were no bugs.

Re:Ppffftt!

Consumers==Unpaid Beta Testers

paying beta testers

How is this possible?

Someone in their QA department needs to be fired. This type of mistake is simply unacceptable, and truly very difficult to believe.

What sort of test plan fails to catch BRICKING THE PC?

Re:How is this possible?

Obviously one with a really high uptime for Windows :)

Re:How is this possible?

Say John, there's a funny thing with our new patch; after the dialogue telling the user that the install was successful and that they should reboot the machine, the machine doesn't actually reboot, it just shuts off and then hangs. What should we do?

Don't tell them to reboot the machine. Problem solved.

Re:How is this possible?

Bricking. That word will have became annoying to me by the end of 2008.

Re:How is this possible?

I think it became annoying to me at the end of November. What's bad is it's usually not the geeks who fall in love with buzzwords.

Re:How is this possible?

BRICKING THE PC?

WTF is with you people? Ever since the Apple made iPhones "bricks", this erroneous use of the term has seeped into our technical vocabulary. People, it's not a brick if it's still usable. When a piece of electronics is really bricked, that means that the ROM is in such an unrecoverable state, that it can't even be flashed with a new working ROM, and needs to be either thrown away, or sent to a factory for repair.

Now, the term bricking is being applied to any piece of electronics or computer equipment that won't boot an OS.

It's not bricked if you can just reinstall or repair Windows and have it work again. It's bricked if you flash a bad ROM BIOS image and now you can't even turn the thing on.

Re:How is this possible?

No, the term "brick" does not change based on your technical experience and "considering" something to be bricked does not make the use of the term correct. Joe Average may refer to his hard drive as "memory" but his use of the term is still inaccurate. If the flash chip on an iPhone is FUBAR'd to the state where you can't even reflash it by any means, it's bricked, whether it's in Joe Average's hands, Steve Job's hands, or Sally Tech's hands. Anything less than rendering a piece of hardware completely inoperable (hardware with the usefulness of a physical brick) is *not* bricked. Now, if the boot.ini removal rendered a hard drive inoperable...

Re:How is this possible?

No. Bricking doesn't refer to the user's ability to operate a given technology, otherwise my grandma could call her new microwave "bricked," which isn't the case. Just because you are too incompetent to use something doesn't mean it isn't useful. A brick is a device that is irreparably broken.

Re:How is this possible?

WTF is with you people? Ever since the Apple made iPhones "bricks", this erroneous use of the term has seeped into our technical vocabulary.

Sheesh... way to brick the discussion...

Re:How is this possible?

I have mod points to give you but you're already at +5. Thank you for giving voice to my frustration over this usage. Imprecise language helps no one. A device is called a brick because it is no more useful than one. If you can fix it, it's just 'broken.'

Re:How is this possible?

No, thats just stupid. A computer with a missing boot.ini is much more useful than a brick. Case in point, you can buy a bare bones computer with no OS for ~ $400. More than your average masonry brick. If people do sell it a state similar to its "bricked state" for more significantly more than a brick, then I would say its not bricked. Because, obviously, its more useful than a brick.

They both made errors.

I suppose both the producers of Eve Online and MS are to blame here. Eve Online for naming a configuration file the same as a Windows system file. And of course MS, for letting any application overwrite such an important system file.

Re:They both made errors.

Likely the users were running the game as administrators, and an administrator would have the necessary rights to overwrite any file on the disk. I don't see how this could be blamed on Microsoft. On Vista you'd get a UAC prompt for trying to write to C:\, but Vista doesn't use a BOOT.INI anyway, so no risk of breaking the system.

Re:They both made errors.

Sad that so many games require Administrator access to run.

--AC

Re:They both made errors.

It certainly is sad that some apps and games need admin privs to run, but this is an installation bug. Of course people are going to install programs as administrator...

Re:They both made errors.

There is nothing that Microsoft could have done to prevent this. Installation of applications to the machine requires administrator privledges, as does installation of drivers. On Vista, there will be a UAC prompt when you start installation and uninstallation, but the process will then run with the full administrator token. Admin's can do what they want on the box. On a *nix system such an installation / uninstallation error would typically nail the system as well unless it was run in a rather full jail, and I am uncertain that jailing the game would have adequately dealt with a process that might install new video drivers. Certainly, most users would have been slammed in either environment.

Microsoft is criticized for its slow release of patches and software. One of the major issues slowing down release is the exhaustive testing passes that software must go through, and they still occasionally miss something. The diversity of configurations in the field is astonishing. This is an issue Apple does not face, as they support an OS for ~ 2 .releases, say 3 years -- and they make all the HW, which limits the diversity. Microsoft supports their stuff for 7 to 10 years (the 9X and ME series were a bit less than this).

Bricking?

Why does the summary say "bricking the machine"? Does the machine become a doorstop that cannot be fixed? Can you not (and this might even be more complicated than necessary, but as a rather inexperienced Windows user, this came to mind first) use a Linux Live CD to boot and edit the necessary files? I DNRTFA, but if it is just an errant backslash, it should be a piece of cake to fix.

Hardly "bricking" IMHO.

Re:Bricking?

As I mentioned in my previous post:

People with one machine and w/o a Linux live CD (probably 90% of windows users) would have a bricked machine barring any outside assistance.

No, they wouldn't. The term "bricked" has very specific connotations. Specifically, that it is not repairable without professional intervention which will probably cost more than the unit itself, thus turning it into a "very expensive brick."

A crashed OS is not a bricking, unless that OS is on firmware or something. If popping in a CD can fix your computer, whether or not you are too stupid to do it yourself, then it's not bricked.

Re:Bricking?

Man, that's serious then. One would have thought that MS would make the Windows CD bootable so that users could gain access to some form of "recovery console".

Partially correct story

Everything the newsstory says is correct, but the issue have been fixed and anyone updating now wont get hit by it.

It is still a momumental fuckup though and the one responsible needs to be kicked in the balls for that kind of stupidity.

It's not bricked!

Dammit! When did "bricking" expand it's meaning from "unbootable under any conditions due to firmware (such as the BIOS) being improperly overwritten" to "Oops, have to pull out the rescue CD"?

Re:It's not bricked!

I think we need to start subclassing the term. It's bricked, but it's recoverable. So it's just a mild inconvenience. A nerf brick? Loose grout?

Apologies

In Parlimentary Republican Iceland, game breaks Windows!

Not sure this is a QA problem...

Things like this can easily happen when your patch doesn't have any CHANGE CONTROL. Imagine this - the patch is ready to go, everyone agrees on it, and then a small group of developers (or maybe even a single developer) decides to make a modification...and implements it badly. It doesn't even go through QA because QA isn't invoked ("oh, that would just delay the release, I'm sure I have it right anyway"). And now you have this.

I know it drives us crazy, I know not every organization implements change control that's sane and logical. But there's a reason it exists!

Re:Bricking? BS! More FUD!

You and I are probably both competent technical people. For my part, I'm an IT director and have done this type of work for 22 years.

Let us assume the two of us, you and I, know more about the Windows registry, bash shell, or using gcc that 98% of the geeks out there. Just for argument's sake.

However, there's a 95% chance that any EVE online player will have the following qualities:

1. Own only one computer.

2. Not be technical.

3. Not read the forums where the information is posted.

4. Be unable to digest and properly utilize the fix information.

So let us re-asses:

It took us, you and I, about 15 seconds to re-write that boot.ini file and *poof* no problem.

That's 5% of the EVE userbase. Add another 20% of the userbase that figures out how to solve the problem. 25% of the people have the fix.

The rest of those poor schlubs are driving to Best Buy to have some incompetent charge them $100 (or whatever)- and that is NOT FUD!!

That my friend is a screwup of massive scope, with huge consequences, because for people who are not geeks- that computer is a "brick".

Eve's boot.ini

The boot.ini for Eve itself contains information about whether you have the "Classic" version or not. The patch that was released for the Classic version did not contain this problem.

The patch released for the "Premium" version does contain this installer error. The change made to the boot.ini is the line that contains this definition, and is changed from Classic to Premium.

It's a very logical problem, easy to fix if you know it, but also incredibly stupid...

Alarmist

If you don't install your games to C: you're fine.
If you've got a 'basic' OS install, e.g. C:\WINDOWS and one partition, you're fine - the boostrap loader guesses, flashes up an error, and boots anyway.
It's a bit of a fubar, but hardly the next apocalypse.

Summary innaccurate/sensational

The result is that anyone who installs the enhanced graphics patch overwrites the windows XP c:\boot.ini file with the EVE client configuration file, bricking the machine on the next boot. Discussion in a couple of forums threads is becoming understandably heated.

This is inaccurate/sensational. I use Windows XP. I play EVE. Last night I installed the "Premium" graphics update and was not affected. The reason seems to be because I have EVE installed on a different drive from my OS drive.

Also, it's not bricking. A repair via install disc will fix it. Booting a linux Live CD (Ubuntu etc) will allow you to re-create your boot.ini.

Bricking == hardware permanently reduced to non-functional status. I.E. only, ever, useful in the future as a brick/paperweight.

Other uses of the term "bricked" or "bricking" are wrong and not supported.

Not a brick, dammit!

A bricked device either to be sent in to the vendor for repairs, or ,as an alternative, can only be revived via special debugging hardware by people with god-like skills in a certain areas.

A blown OS is not, and never ever will be a brick. Get your terminology straight for once. Wikipedia explains rather nicely the nature of real "brick".