Bethesda Investigates Shivering Isles Bug

Gamespot reports on a glitch in the recently-released Shivering Isles expansion to Elder Scrolls IV: Oblivion. It's unclear at this time if the Xbox 360 version of the title also has it, but on the PC side of things a game-breaking snafu can arise after 50-120 hours of gameplay post-Isles installation. "The bug apparently starts affecting the game as soon as the expansion pack is installed for the PC version of Shivering Isles. The problem arises because the game generates a huge number of identification numbers internally for objects, and once the allotted space for those numbers becomes exhausted, newly created objects will disappear from the gameworld and the game could simply crash ... It appears that the more frames per second the game runs at, the faster that space of identification numbers fills up."

Broken script

[pdbaby]

Seems it's a broken NPC script with 6 of the new additions (all generic guard characters, amusingly). Time to move to 64bit? :-P

Re:

[mikkelm]

I hope you're joking. Somehow I doubt that this is caused by exhaustion of addresses in a 32 bit field. If it was, your Oblivion savegames would be above 16GiB each.

Re:Broken script

[DogFacedJo]

    It is caused by the exaustion of IDs in the low 24 bits of the ID. The field was intended to roll over back to 0 since the majority of the IDs are only needed temporarily. Unfortunatly they bugged the implementation, failing to preserve the 8 high bits (normally FF) in all rollover cases thus generating much internal confusion.
    The problem was exascerbated by the new guard scripts which really burned through the IDs fast.

Re:

[SatanicPuppy]

Pssh, they used some dinky variable type like "short", assuming that they'd never need more. If they'd used 32, they'd be fine right now.

Either way it's sloppy. You should never have anything in your code that just keeps growing every time you run the code, unless you're keeping time or something...Because this issue is continuing after they stop and start the game...It's saving this crap on the hard drive somewhere, and that's just damn sloppy.

Re:

[SanityInAnarchy]

You should never have anything in your code that just keeps growing every time you run the code, unless you're keeping time or something...

Or you're simulating something which has the ability to just keep growing. It's not necessarily a memory leak if it's actually using that.

And by the way, if you are keeping time, note that we won't run into trouble for at least a few decades, by which point we'll all be on 64-bit, and it should be trivial to switch to a 64-bit integer for timekeeping.

It's saving this

News from the future... today!

[Clever7Devil]

This happens in World of Warcraft: The Burning Crusade as well. It occurs when a character casts too many beneficial spells in a short period of time. Blizzard is reporting that it is a Buffer overload.

Deja vu

[hansamurai]

Hey, this kinda reminds me of the time Slashdot hit the comment limit [slashdot.org] in its database.

Ha!

[SatanicPuppy]

So the number field they chose to represent the key for these objects was too small, so after a period of time it fills up and causes unstable behavior?

Who says programmers never learned from Y2K?

I did some Y2K work in Y1.998K, and it's stuck with me...All my auto-numbering keys are as big as I can make 'em. Any number that grows, I figure out the largest possible size that number could be, and then put it in a datatype that's capable of hold a number at least twice as big.

Sure it's a waste of space, but on

Re:

[remembertomorrow]

__int64/BIGINT for the win!

Re:

[illegalcortex]

It's not just a waste of space, it's a waste of time. Using certain data types can be much faster, especially where graphics are concerned. While you don't want screw something up by skimping, picking the right data type for the situation is the way you get the highest performance from your code. Games would be a case where this matters a lot.

Re:

[SatanicPuppy]

But this isn't graphics code...This is unique object identifiers. With graphics you know how big all the numbers are going to be! You set hard limits on all your rendering, and you're done. If your code isn't set to render past a certain point, it just won't...If you're set up to only process X vectors, it doesn't matter if the system is capable of more...And moreover, nothing grows in graphics processing. You process the same level all the way through...If the system has to do more work with a series of f

Re:

[illegalcortex]

Sigh. I wasn't saying this WAS graphics code. I thought that was fairly clear. I was just making the point to you that choosing your variable size isn't just about size, it's about performance. CPU architecture handles certain variable sizes differently. This means different variables types will actually affect the performance of your app, even if it has nothing to do with the graphics.

Also, if you read the post right after yours, you'll see that this wasn't even the problem. The problem in the weird

Re:

[SatanicPuppy]

I did read it. I even responded to it before I responded to you.

Prefixing, in my mind, is the wrong word for what they did...They started numbering at FF000000 or whatever, assuming it'd never roll over. So first off, that's just dumb. You should never start an incrementing counter from anything but zero...To point at your original argument, they used way the hell more space than they needed, because, in their minds, all they were ever going to need was FFFFFF, not FFFFFFFF.

Second, it's still a size issue.

Re:

[illegalcortex]

I wrote:

Also, if you read the post right after yours, you'll see that this wasn't even the problem ... Except you should already know this because you replied to that post BEFORE you replied to this one.

And you replied:

I did read it. I even responded to it before I responded to you.

It's at this point I realize you are not really reading my posts. You must be reading them somewhat, but maybe "skim" is a better word. But this reply makes me understand why you're absolutely not getting what I'm saying about

Re:

[SatanicPuppy]

Well, yea, I was skimming. But looking back, I don't think I was that far off, though I did make some leaps from what you said...In my mind, all the interesting performance issues with games are in the area of graphics...A 64 bit object identifier wouldn't have really made a dent in processing, especially for a game like Oblivion, which has so much positional data to keep track of, so I just leapt to the (to me) interesting bit, and ignored the back end stuff which you were probably talking about.

I understa

Re:

[geekoid]

On some systems thats fine, on many types of programming you need to control all the information, and keep it tight. Like..say.. Graphics for example.

This bug is clearly due to amaturish code.

Seen it Myself

[DG]

I'm armpit-deep in Oblivion these days, and Niner bought me the SI expansion for my birthday.

When I saw the announcement about the bug, I tested it for myself, and it is very real. Happily, there is a community-produced script patch that disables the problem.

Interestingly, the root cause of the problem isn't so much the broken scripts that consume ObjectIDs, but rather that some intelligence was baked into the ID instead of it being just a raw counter.

The ObjectID is prefixed with "FF" and the remaining bytes are the counter values. When the counter hits FFFFFFFF it rolls over, and the "FF" prefix no longer applies. *That* is the problem - the game code no longer recognizes the ObjectID as valid without the FF prefix.

It seems that the code that generates the next ObjectID is smart enough to skip IDs that have been assigned; hacks that reset the ObjectID counter back to FF000000 appear to do the right thing. If the counter had no prefix, the bug wouldn't affect the game - the counter would roll over, but any objects that had been around since the start of the game (with low ObjectIDs) would be properly skipped and all would be well. Unless you managed to have FFFFFFFF objects extant in the game world, there'd never be a way to run out.

Happily, my counter was at FF4xxxxx so my game save is OK. I feel for the guys who discovered they were at FFFxxxxx.

DG

Re:Seen it Myself

[SatanicPuppy]

Wow, that's incredibly short sighted of them...I've seen crap like that before, working on databases built by amateurs. Like a lookup table that doesn't have a "flag" to tell the query which dataset the entries belong to, but instead a situation where the original designer just incremented the key by an arbitrary amount, e.g Rate codes are 1-999, product_codes are 1000-1999, etc, so when the 1000th rate code is added, it writes over the first product code, and chaos ensues.

You should never make the "size" of a variable part of it's identifying factor.

Re:

[Abcd1234]

You should never make the "size" of a variable part of it's identifying factor.

That's just silly. People do this all the time for a variety of reasons (often for space concerns). Hell, IP address are just 32-bit numbers, and if the lower 8-bits of an address roll over, the address is in a different subnet. Sounds like a violation of your rule to me. And I'm sure I could come up with myriad other examples.

What you shouldn't do is make such an assumption *unreasonable* (obviously 1000 codes is ridiculousl

Re:

[SatanicPuppy]

Well, IPv4 would be considered a bad example by a lot of people, because of how quickly we ran into problems with the size of the address space.

The thing with IP addresses is that they don't just keep incrementing...There is a hard limit on how many you get, even for an internal subnet, though it's unlikely anyone would ever max one out. If they did just keep incrementing, the internet would flop because the system would be re-assigning addresses to different people all the time. As a semi-relevant aside,

Official patch (beta though) is available already

[DogFacedJo]

A tested patch is clearly only a short way away.
    This story broke about 4 or 5 days ago... and the longer you put off patching the more likely the problem is to hit.

http://www.elderscrolls.com/downloads/updates_patc hnotes12.htm [elderscrolls.com]

Re:Official patch

[DogFacedJo]

I feel strangely dirty...

    As if I somehow was paid for something that should have been freely given.

        Next time I leave out the link. ;}

Re:

[ultranova]

It seems that the code that generates the next ObjectID is smart enough to skip IDs that have been assigned; hacks that reset the ObjectID counter back to FF000000 appear to do the right thing. If the counter had no prefix, the bug wouldn't affect the game - the counter would roll over, but any objects that had been around since the start of the game (with low ObjectIDs) would be properly skipped and all would be well. Unless you managed to have FFFFFFFF objects extant in the game world, there'd never be a

fills up faster?

[metalpres]

so it fills up faster and crashes sooner the higher the fps? well then i should be able to play the game for about 500 hours cause i cant get more then 15fps in that game... :(

Re:

[darkrowan]

300 hours, roughly I guess. Here's better data on how much time you have left vs both current reference ID and framerate
http://www.uesp.net/wiki/Shivering:Reference_Bug#R emaining_Playtime [uesp.net]

Re:

[jafuser]

"The basis of the bug is that an internal "space" of identification numbers for objects created in the game (e.g., a dropped arrow, a new creature, etc.) becomes exhausted"

Hah, I used to collect arrows by dodging them. Probably another reason I hit the bug so early in the pre-expansion game. =P

Re:

[Loligo]

>Hah, I used to collect arrows by dodging them.

Guess that beats collecting arrows by NOT dodging them...

  -l

Re:

[jafuser]

You can do that too. If I recall correctly, I think there is a 50% chance that getting hit by an arrow puts it directly into your inventory =)

Bethesda and the Time Bombs

[Deorus]

This sounds almost like the Slow Animations [bethsoft.com] bug that happens after roughly 200 hours of gameplay and has never been fixed. People playing on the PC can hack their saves to reset a 32bit float value to temporarily work around the problem, though those with the Xbox (and probably PS3 too) version of the game are SOL.

Re:

[Mongoose]

PS3 version of SI isn't out yet. Maybe it's a good thing. The PS3 version is superior to the 360 version with the new shaders and additional content. Yes, buy the game for every platform. ;)

Re:

[MBraynard]

I understand the 360 will get a shader update.

Re:

[iainl]

The 360 got its shader update the Thursday or Friday before Shivering Isles went up (so before the Live downtime the other week, if that helps you). If you've played the game since then, that was why it had to perform an update.

I think it makes a big difference, but others don't think it makes as much of one. The main change is in outdoor areas - the Level-Of-Detail method for bringing foliage in from a distance is much smoother.

For the record, it still doesn't make things look quite as nice as the PS3, and

Re:

[jafuser]

Hah, this sounds like the exact same glitch, and explains what was going wrong.

I stopped playing Oblivion only a couple months after release because of this. I had (for the time) a relatively fast computer and I loved to collect and gather tons of herbs and junk. I started having this problem at around 200 hours into the game.

I always suspected that the amount of stuff I was collecting probably had some co-relation to the "slow animations" bug.

It was a very elusive sort of problem because nothing seemed t

Quick!

[Some_Llama]

Somebody should submit this to http://www.worsethanfailure.com/ [worsethanfailure.com]

Oblivion = too many bugs...

[necro2607]

Ouch, Bethesda seems to be making a bad name for themselves with Oblivion, in terms of software bugs. I don't mean to "troll" but my girlfriend's game has been screwed up multiple times, including the health bar no longer increasing, or the game bugging in such a manner that a quest could never be completed. We searched forums and contacted support about these issues (among others), with no solution ever found. Luckily I haven't encountered such extreme problems myself, but I have noticed small bugs from time to time. Regardless, such showstoppers as mentioned above are pretty unbelievable for a $70+ game of such supposedly high stature.

Re:Oblivion = too many bugs...

[sholden]

it's a daggerfall sequel. They're still striving to meet that bug count standard.

Re:

[necro2607]

That's lame, though. Just because their previous stuff was buggy as hell doesn't make it OK for them to keep doing so... I have pretty reasonable standards, I'll tolerate some bugs here and there, but I don't consider it acceptable to allow total show-stopper bugs that render the game unplayable (your health never going up pretty much makes it impossible to continue playing the game)...

Re:

[Shados]

Thats probably why PC gaming is slowing down a lot more than because of MMOs... between games like Oblivion and Neverwinter Nights 2...these games were released in a state that could barely be qualified as alpha by console standards. I am and always was a PC game fan, but this is getting out of hands.

Re:

[__aamkky7574]

Thats probably why PC gaming is slowing down a lot more than because of MMOs... between games like Oblivion and Neverwinter Nights 2...these games were released in a state that could barely be qualified as alpha by console standards.

Ummm, you do know that the XBox 360 version has the exact same bugs as the PC version, right? And at least on the PC, you can avail of an unofficial patch which includes about a 1000 bugfixes. No such patch for the XBox 360. I can't see why you think that console programming inv

Re:

[ultracool]

Since installing Oblivion, I've had real problems with games crashing under Windows. Oblivion would crash every 15 minutes or so, and nasty crashes too! Some of them would even reboot my machine! It's too frustrating to play now. I'm going to have a go installing it under Wine. I won't be able to have HDR, but at least the game might be playable. Like Windows needs something else to make it more unstable...

Re:

[Khanstant]

They're features to encourage you to play better games.

Re:

[Joker1980]

Couldent agree with you more, but its not only Oblivion. Morrowind was also a buggy mess. Dont get me wrong i enjoyed both immensly but as software becomes more complex testing is going to have to become a higher priority for all software.

People are actually still playing this POS?

[Hazelnut]

Wow. It's a dumbed down, albeit pretty, game. It's hardly much of an RPG, and I got bored after 15 hours when I realised how utterly shallow and moronic the game was. The patronising inbuilt quest walkthrough comments and complete lack of choice is most situations is absolutely unbelievable. It's a crying shame the direction the current bunch of idiots at Bethseda, lead by Todd 'boobies' Howard, has taken the ES series. I loved Daggerfall, and Morrowind was pretty good - Oblivion should have taken the stron

Makes me wonder...

[A_Non_Moose]

if Bethesda ever figured out the reason for the "Stutter Bug"?

Common factor seemed to be 200+ hours of gameplay and the animations for doors (fire, too) would
"Stutter" so badly in ruins it was impossible to do anything.

Thank $deity for the mod community and someone with a savegame 30'ish seconds before the bug.

Sounds very similar to this particular bug, but where I stopped playing was a month or so
after that bug was fixed by the mod community.

Several long continuous threads might still exist on elderscrolls

Bethesda games have always

[geekoid]

been chocked full of bugs, or hugh areas of nothing.

And I mean serious bugs. They should get a QA specialty consultant to go over there QA process.
They could be good games, but I know when I see a bethesda game there will be bugs that should have been showstoppers and will crash the game, if not the computer.

Exactly why I haven't picked up one of there games in a number of years.