Slashdot Log In
The Truth About Last Year's Xbox 360 Recall
Posted by
kdawson
on Tue Jun 10, 2008 06:03 PM
from the value-for-the-money dept.
from the value-for-the-money dept.
chrplace forwards an article in which Gartner's Brian Lewis offers his perspective on what led to last year's Xbox 360 recall. Lewis says it happened because Microsoft wanted to avoid an ASIC vendor. "Microsoft designed the graphic chip on its own, cut a traditional ASIC vendor out of the process, and went straight to Taiwan Semiconductor Manufacturing Co. Ltd., he explained. But in the end, by going cheap — hoping to save tens of millions of dollars in ASIC design costs, Microsoft ended up paying more than $1 billion for its Xbox 360 recall. To fix the problem, Microsoft went back to an unnamed ASIC vendor based in the United States and redesigned the chip, Lewis added. (Based on a previous report, the ASIC vendor is most likely the former ATI Technologies, now part of AMD.)"
Related Stories
This discussion has been archived.
No new comments can be posted.
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
Full
Abbreviated
Hidden
Loading... please wait.
I'm Shocked.... (Score:5, Funny)
Re:I'm Shocked.... (Score:5, Funny)
The Red Rings were a FEATURE! I tell ya!
Parent
Re:I'm Shocked.... (Score:5, Funny)
Parent
Re:I'm Shocked.... (Score:4, Funny)
Parent
Re:I'm Shocked.... (Score:5, Funny)
Parent
Re:I'm Shocked.... (Score:5, Funny)
Parent
Re:I'm Shocked.... (Score:5, Funny)
Parent
yes, go cheap, that's the way (Score:4, Interesting)
Re:yes, go cheap, that's the way (Score:5, Insightful)
MBAs are good in cutting corners in traditional businesses, but generally have no understanding of technology risks....
Parent
Re:yes, go cheap, that's the way (Score:5, Funny)
Parent
Re:yes, go cheap, that's the way (Score:4, Funny)
Parent
Re: (Score:3, Insightful)
Re:yes, go cheap, that's the way (Score:5, Insightful)
You can have business savvy and technological expertise, but it's a roundabout path through today's educational system if you're not teaching yourself at least one. And I think we all know the proportion of people who are capable of serious self-education.
Parent
Re:yes, go cheap, that's the way (Score:4, Insightful)
And for balance the problem I have with engineers as managers is that it's possible to learn the people skills stuff but you have to understand why it's important and want to do learn it. It's all too easy to stay in the comfort zone where you basically sit in a dark corner somewhere and write code if that's what you enjoy rather than forcing yourself to talk to people.
Parent
Re:yes, go cheap, that's the way (Score:5, Insightful)
I don't think you're getting it. Cutting costs is one thing. Cutting corners is another. Cutting costs is fine, but cutting corners implies the product is worse off because of it. Few engineers would say "It'd be cheaper to roll our own graphics chip," because they realize the immense technical challenges involved. Few MBAs are likely to understand that, however.
There's a big difference between what you just said and what the OP said. Nobody said MBAs can't be tech savvy. However, the fact of the matter is, most of them aren't.
Also, just to be pedantic, having an MBA has little to do with having business savvy.
Parent
Re:yes, go cheap, that's the way (Score:5, Informative)
The mistake seemed to be to let Microsoft's in house group do this rather than outsourcing.
But you've got to remember this is an article in EEtimes from an analyst with an agenda
http://www.eetimes.com/news/latest/showArticle.jhtml;jsessionid=51TYZYXYRWUZUQSNDLSCKHA?articleID=208403010 [eetimes.com]
"System OEMs have no business designing ASICs any longer," said Lewis. The reality is that system companies are finding it hard to do enough ASIC designs to keep in-house design teams employed.
Basically he's trying to create business for ASIC design houses by telling people that putting a bunch of licensed IP onto a chip is rocket science and they shouldn't try to do it in house.
Is it really? I honestly don't know. I suspect it depends a lot on the quality of the in house people and the quality of the ASIC design house.
And it depends on what you're trying to do. In the embedded area lots of companies much smaller than Microsoft put an processor and a bunch of their own peripherals onto a chip and it works. I guess that console or PC graphics cores use a lot more power than that. But I don't know if "an ASIC design house" would have done a better job than Microsoft's ASIC group.
Or more to the point, maybe a $1B recall is the price you pay for learning about this stuff. Microsoft can afford it obviously and it will influence how the successor to the XBox360 is done. Whether they hire more engineers and do it in house or outsource it is a business decision it seems. I guess the in house people and the design house will both try to argue for the best option from their point of view and some manager will decide.
But if you're a cash rich company then the bias will be to try to do as much as possible in house, because that gives you more freedom to value engineer later.
Parent
Re:yes, go cheap, that's the way (Score:5, Insightful)
It is true. You should not unnecessarily muck with VHDL/Verilog and 3rd party cores even if you work with an FPGA. This will not kill you, but it will make you poorer. HDLs are notoriously kludgy, and it takes a lot of effort to do it right. Proprietary cores rarely work as documented, and you have no visibility into them. When multiple cores are used, it's one large fingerpointing game between vendors. And you need to have good, experienced HDL coders. And you need to have all the tools, they cost big bucks.
But that's with mere FPGAs, where you can update your design whenever you wish. However here they are talking about ASICs - where all the wiring is done with masks when the IC is made. You'd have to be certifiably mad to even think about a casual design like this. ASIC designs are done by very competent teams, using "10% coding / 90% verification" time allocation, because you can't afford /any/ mistakes. And even then you make mistakes; but experienced teams with good tools make those mistakes smaller, and they call them "errata" - something that is not right but can be worked around. When you make the F0 0F bug, though, you trash the whole run.
So Microsoft risked a lot when it went for an in-house design. I am not surprised that they failed. They should have counted all the successful 3D video companies on the market and asked themselves why there are so few, and why top gaming cards cost so much.
But if you're a cash rich company then the bias will be to try to do as much as possible in house, because that gives you more freedom to value engineer later.
I am not MS, but I don't really see much business value in rolling your own video controller. More likely the NIH syndrome kicked in, or some people were overly concerned about their job security.
Parent
Re:yes, go cheap, that's the way (Score:5, Insightful)
That's true, but, if the did go to an ASIC vendor they could have got a contract indemnifying them from taking losses when the chip turned out to be flawed. By doing the chip design themselves, they saved a little bit of costs, but also took on all the risks of having a bad design.
That's what the parent poster is alluding to. A manager with experience in technology would have understood that, while designing your own chip might have been cheaper, it would have also introduced significant downside risk, which ought to have been factored into the equation. Farming the chip design out to a third party, while more expensive in the short term, would have entailed less long-term risk.
Parent
Re:yes, go cheap, that's the way (Score:5, Funny)
Parent
Going cheap may well be the sensible way... (Score:3, Insightful)
Consider: would you rather spend $10M on a platform that may flop and not make a dime
OR
Spend $1B on a platform that has made multi-billions.
Re:Going cheap may well be the sensible way... (Score:4, Insightful)
Parent
Re:Going cheap may well be the sensible way... (Score:4, Insightful)
Parent
Re:yes, go cheap, that's the way (Score:5, Insightful)
Parent
Bleh... (Score:4, Insightful)
When I read that I pictured Ballmer: (Score:5, Funny)
Re:When I read that I pictured Ballmer: (Score:4, Funny)
You picture Ballmer?
Your therapy bill must be astronomical
Probably not too far off the truth, but still...you think they'd learn.
Parent
Re:When I read that I pictured Ballmer: (Score:5, Interesting)
Parent
Re:When I read that I pictured Ballmer: (Score:5, Funny)
Parent
Chickens are coming home to roost... (Score:5, Interesting)
Or is it all just a hoax? [fugue.com]
Hope not.
Re:Chickens are coming home to roost... (Score:5, Funny)
Parent
Another Talisman CF (Score:5, Interesting)
Never, and I say NEVER let a bunch of software engineers try to design a hardware chip. This was the biggest CF I'd seen in all my years (30+) as a chip designer. That they did it again, and with such stupidity again is no friggin surprise.
It is not that software engineers should not be involved, of course they should but when they drive the architecture in complete void of any practical chip design constraints..... and continually refuse to listen to any reason from the hardware designers..... well as they say, garbage in, garbage out.
Re:Another Talisman CF (Score:5, Insightful)
I suppose if we can all agree to stay out of the other guy's yard, we can get along. You do hardware, I'll do software.
-g.
Parent
Re:Another Talisman CF (Score:5, Interesting)
I testify, Brother, I TESTIFY!
30 Years ago, I ended up in therapy (literally) after dealing with an assembly program written by a hardware guy. The program emulated a CDC communications protocol that was originally done in hardware. This was on a Cincinnati Milacron 2200B, a machine that had both variable instruction length and variable data length. The hardware guy had implemented the protocol state changes by putting a label *on the address portion* of jump statements (he did this in 50 different places in the program) and then in some other area of the code he would change where the jump branched to next time through. It bordered on an implementation of the mythical COME FROM instruction. Of course, there was zero documentation and almost zero comments.
After one marathon debugging session I was so frustrated I was in tears. My manager came in and wanted to know what the problem was. I gave him the listing and left to walk around the building a few times. When I came back, he told me that it was, hands down, the worst piece of crap he had seen in 20 years. He had me rewrite it from scratch, which I did over a long weekend.
The program's name was RIP/TIP (Receive Interrupt Processor/Transmit Interrupt Processor) and I was in therapy for most of a year. (There were a few other issues, but this was the bale of hay that made me snap.)
Parent
Re:Another Talisman CF (Score:4, Informative)
I can tell you first hand that a lot of the people on the Xbox hardware team a extremely talented HARDWARE specialists. The way you talk you would think MS locked a bunch of IE developers in a room and didnt let them out until they had designed the chip.
And as for the argument of 'well if they are so talented, why is the chip such a POS?', it is not only software engineers that design shitty hardware. Look at AMD, with the TLB defect in the Phenom chips, is that the fault of the software engineers?
This response may be overkill, but somehow you were modded +5 interesting, but you completely miss the point.
Parent
Re:Another Talisman CF (Score:5, Interesting)
I am a person that designs both hardware, and software, but not chips, At the risk of talking outside of my expertise, I will have a go at answering your question.
Firstly, there are things that software people really like, but it is often better to not do them in hardware. This category contains things like Read/Write I/O registers. From a software point of view, they are nice, but they can double your gate count. They can also increase your capacitive bus loading. DAC and ADC designs can also be affected this way. A software person might use a proper ADC and expect proper ADC registered results. A hardware person might select a resistor, capacitor, a voltage comparitor, and a couple of spare I/O pins. The cheesy R/C approach may save the hardware design from a whole slew of problems including cost. A software person may opt for a synchronous logic approach with all registers clocked every clock cycle. The hardware designer may opt for a much more asynchronous approach, that minimizes the number of clocked registers. This reduces power consumption, and potentially the number of registers too. Often the hardware designer will consider thermal, cost, electrical layout issues as part of his design process. The software person will not be as familiar with how to design a good circuit board and chip design in a cost-effective manner. A good software engineer can learn all of this material with time, but the hardware engineers will do them naturally.
The second category of problems is tools. The modern chip designer is working with a fairly advanced set of tools that the software person is likely to be quite unfamiliar with. This starts with the IC design tools, which are quite specialized. It ends with the hardware engineering tools. Have you ever X-Rayed a circuit board to analyze the cracks in the Ball Grid Array where it bonds to the circuit board? Are you familiar with thermal issues, and thermal images? How about EMI test results? Modern IC package design limitations? A good team of engineers will be familiar with these tools, and know how to use them to get good results.
The third category of problems is mistakes from inexperience, or lack of experience in the correct field. I work with industrial electronics. I think from an industrial point of view. What happens when someone attaches 600 (VAC) to the ground wire of the computer? What happens to the remote sensors when the plant gets hit by lightening? In IC design, there are some known gray areas too. Does the chip reset properly on power up? Do metastable, astable, or self-oscillating states exist in the IC design? Can the chip survive with no cooling? Does the chip have an overtemp shutdown function? What happens if someone starts the chip up in sub-zero weather? Do the analog electronics have sufficient electrical separation from the digital electronics, while avoiding nasty things like ESD latchup conditions?
I've completed chip design courses before, but have never had to design a modern production gate array design. As a person that has done both software and hardware, I know that my skills are not good enough for the most modern IC design processes. My limit is FPGA work, and my preference is clever opto-isolation, power semiconductor, TTL and micro-proccessor based circuits. In analog, my expertise in analog is industrial sensing and survivability. You have to know where your field of expertise is, and what your limits are.
Parent
Re:Another Talisman CF (Score:5, Insightful)
Parent
Re:Another Talisman CF (Score:4, Insightful)
Well... there's "real" software engineering too...stuff involving resource deadlock, race conditions, critical section synchronization, in applications like virtual memory management, network protocols, time sync, file systems, security, fault tolerance, etc that are subject to all sorts of 'physical reality nastiness'.
Its not all wizards and automatic code completion you know.
Parent
Re:Another Talisman CF (Score:5, Funny)
Parent
Ridiculous (Score:5, Informative)
Furthermore, the recall was for overheating in general which -- though unquestionably affected by the GPU -- is a more comprehensive system design failure, not just a single component. (Look at the stability success they have had simply by reducing the size of the CPU.)
I'm looking forward to "Jasper", the code name for the next XBOX 360 mother board that will include a 65 nanometer graphics chip, smaller memory chips and HOPEFULLY a price reduction.
Vote parent up (Score:5, Insightful)
Years before the xbox360 has been released ATI was already announced as the system parter for the GPU. No "secret unnamed ASIC vendor" anywhere.
The recall, again, was thermal problems.
Do you really think a completely different GPU by a completely different company could have been designed in a year _and_ totally compatible with the original one?
Parent
Re:Vote parent up (Score:5, Informative)
Parent
Re:Vote parent up (Score:4, Informative)
Parent
What's going on..... (Score:5, Informative)
Lewis seems to be just plain wrong, which is kind of upsetting for "chief researcher" at a firm like Gartner, especially when the correct information is freely available.
While the cooling solution for the GPU is the likely cause of most of the failures, that's not necessarily the GPU's fault, or ATI's, especially for a fault so widespread.
Re:What's going on..... (Score:5, Interesting)
Everything I've ever heard as a "Gartner opinion" got one of two reactions from me:
1. Well duh.
2. No, that's obviously wrong.
Looks like this is #2.
Parent
Re:What's going on..... (Score:5, Informative)
Parent
What Recall? (Score:4, Insightful)
True to
News for Nerds.... Stuff that may or may not be true...
this doesn't seem accurate, it was solderability (Score:5, Interesting)
The problem wasn't any chip at all. It wasn't even heat. The problem was the chips were not soldered to the board.
http://www.bunniestudios.com/blog/?p=223 [bunniestudios.com]
Doesn't matter who designed or made the chips. If they aren't soldered down, they won't work. And that's what the problem was. That's why X-clamps (mostly) work.
Heat is semi-tangential. If the chip is soldered down, heat won't pop it off and if it isn't soldered, any kind of movement will break it loose, even when cold. This is how MS could ship you replacement units that were RRoD out of the box. They were fine before they were shipped and were broken loose during shipping.
Most of the problem appears to be solderability problems, not a problem with chip design or manufacturing.
Re:SLASHDOT SUX0RZ (Score:5, Funny)
Parent
Re:More info please (Score:4, Informative)
The General Hardware Failure error could be caused by cold soldering. The added mass of the CSP chips (including the GPU and CPU) resists heat flow that allows proper soldering of the lead-free solders underneath the motherboard.
Another General Hardware Failure is shown by the ring of light flashing one red light, and an error code E 74. This too renders the Xbox unusable.
The Nyko Intercooler has also been reported to have caused a general hardware failure in a number of consoles, as well as scorching of the power AC input.
An update patch released on November 1, 2006 was reported to "brick" consoles, rendering them useless.
In June 2008, the EE Times reports the problems may have started in a graphic chip.
Parent
Re:Some Facts... (Score:4, Interesting)
The PS3 COULD run it in 360-resolution, but it might have to sacrifice some of those filters and special effects. I'd rather have a special effect laden game run at slightly lower resolution myself, as long as its hard to notice.
Parent