Eight PS3 'Supercomputer' Ponders Gravity Waves

Jamie found a story about a inexpensive supercomputer being used by an astrophysicist to research gravity waves. The interesting bit is that the system is built using 8 PS3s. Since nobody is actually playing games on the system, it makes sense to use them for research projects like this, but I really wonder now what is defining 'Supercomputer'... I mean, a hundred PS3s sure, but 8? I think we are de-valuing the meaning of the word 'super' :)

Re:8 systems x 8 cores =

[Kupek]

It's only six cores available to Linux per Cell processor on a PS3. One is reserved for the Game OS, and one is disabled to achieve a higher yield on fabrication. (The Game OS is always running, since Linux actually runs on top of a hypervisor.)

"We Report. We Decide."

[Intellectual Elitist]

> Since nobody is actually playing games on the system

You can always count on Slashdot for a fair and balanced presentation of information.

Funny that I've bought 4 disc-based games and at least one downloadable game since the beginning of July, and have been using my PS3 almost exclusively for gaming since then. I'll be buying at least 4 more games before the end of the year, too.

Sorry to burst your bubble, but the PS3 game drought has been over for a while now...

Re:Inexpensive, eh?

[Bandman]

It's not just in the summary; read the article, it dishes out the abuse

G4 was a supercomputer ... at the time.

[oneiros27]

It met the government's definition of super computer at the time. (1.5Gflops ... well, technically, 1500MTOPS).

The designation is part of the "Dual-Use" restrictions on exports (basically, things which could be used for both military and non-military applications).

The 1Gflop threshold was set as the necessary processing power to calculate balistic trajectories for missile systems.

I can't find the documentation, but my understanding is that the current threshold is 190Gflop (since Jan 2002).

Only 256 Megs of RAM

[hweimer]

The CPU power of the PS3 is indeed very impressive, however, for most real-world supercomputing tasks the 256 MB RAM per node are way too low. One Gig per core should be the minimum, meaning you would have to increase the amount of RAM in the PS3 by a factor of 24.

Definition of a Super Computer?

[Midnight Thunder]

Looking at this page:

    http://www.answers.com/topic/supercomputer?cat=biz-fin [answers.com]

they define a 'supercomputer' as being "A mainframe computer that is among the largest, fastest, or most powerful of those available at a given time". This is suitably vague, since the point of reference changes all the time. On the other hand there is no point of reference in the definition. For example, does it have to be in the top 100 or 100x more powerful than the current top of the line PC? Without a suitable reference point anyone could call their cluster amongst, the "largest, fastest or most powerful".

1.2 TFlops

[rockmuelle]

8 PS3s gives you 1.2 teraflops of single-precision performance or a similar number if you stick to integer operations (6 SPUs/PS3 gives ~150 GFlops). 1.2 teraflops is a supercomputer in my book.

Using Jack Dongerra's single-precision algorithms that do half the work in single and the other half in double precision, you can maintain a high level of performance and precision. And, the unique architecture of the Cell opens up some interesting algorithmic research issues, allowing scientists to publish twice for the same work: once for the science results, once for the computer science results. :)

On the flip side, the Gigabit ethernet on the PS3s isn't really 1GB - the PPU can barely keep up. So, extra care must be taken around communication points. And, a similar Intel/AMD-based rack would run about $20k and is much easier to develop for, so if your labor is expensive (i.e., you're not in academia), PS3 clusters may not make much sense.

-Chris

Re:devaluing super

[Ant P.]

You mean like this? [ibm.com]

Re:8 systems x 8 cores =

[Impy the Impiuos Imp]

It's really about pipelining massive amounts of data and calculations. While each calculation may take a noticeable finite amount of time, if each step of the calculation is lightning fast, then the average becomes lightning fast.

This is like an assembly line where it may take 8 hours to build a car, but if the longest stage of the assembly line is 30 seconds, then a car is "made" every 30 seconds as one rolls off the end.

Supercomputers try to use this many-stage pipelining for everything from reading in the data into gigantic local vector registers, then providing operands to operate on the gigantic vector of numbers as a whole, then read it back out pipelined.

A game machine makes for a good supercomputer because the I/O is designed to be fast so it can load up bitmaps quickly, and it has massive parallelism to crunch billions of numbers, all in the same way. If you can write code to take advantage of this (especially if you can split it up amongst several such game machines) bingo!

Re:Inexpensive, eh?

[smussman]

Bullshit. Sony is a hardware manufacturer. They make the whole thing from end to end. They create the price and take all the profit.

http://en.wikipedia.org/wiki/PS3#Sales_and_pricing [wikipedia.org]
Summary of relevant parts of article:
Sony was losing at least $240 per console at launch.
With new manufacturing techniques, etc, they're losing somewhere under $100 dollars.

Either way, they're losing money.

Re:Personal Supercompter?

[for_usenet]

Check out Mercury [mc.com] systems. These aren't exactly for home use, but they are shipping Cell + Linux computer systems. However, given the economics, it might just be better to go with a PS3, if you can live with the memory limitations. There may be other companies, but checking at TerraSoft [terrasoftsolutions.com] (one vendor for PowerPC Linux software and hardware) takes you right back to IBM and Mercury hardware.

Re:devaluing super

[mihalis]

Wouldn't it rather be IBM that might offer this, since they actually make the cell?

Yes, actually I think you are correct. If I recall correctly it's Sony, IBM and Toshiba in the cell consortium, and the most ovious vendor of a "compute-node Cell module" would indeed be IBM, not Sony, good point.

By the way, I had a typo, it would not be an "HPC PSP3" of course, the Cell is way too hot and power hungry! Although ... of course with sufficient shrinks and price reductions the current Cell might well one day be in a portable game console. Then we could have another round of speculation on personal clusters. I love the "wheel of reincarnation" in digital technology!

Re:Not surprising...

[Rayonic]

Under PS3 linux, you're locked out of low-level access to the GPU. Which is a shame, since it's the GPU that does most of the heavy lifting for the Folding@Home app.

Gravitational, not gravity...

[Anonymous Coward]

Gravity waves are the atmospheric phenomena, gravitational waves are the astrophysical ones.

Re:Supercomputer is a term that changes

[Stefanwulf]

Sony's estimate of 1 teraflop per unit is more than a little bit optimistic, and also counts all the operations performed by the graphics system, which the average researcher won't have access to. For a slightly more realistic estimate of what 8 ps3's running linux can do, I'd point you towards http://www.terrasoftsolutions.com/store/index.php?submit=software&submitimg%5Bhardware%5D%5Bsolutions%5D=1, [terrasoftsolutions.com] who sell turnkey ps3 clusters. They are claiming that 8 units together break the theoretical teraflop mark, which seems more realistic to me. While that isn't among the fastest computers in the world by any stretch, it's still solidly in the realm of what I'd call a supercomputer.

The cell is a fantastic piece of equipment - Dr. Dobb's has what I think is an excellent analysis of the kinds of performance benefits that it offers at http://www.ddj.com/hpc-high-performance-computing/197801624 [ddj.com]. I'm currently running one at home in a PS3 (for neural networks that drive an AIBO - I love Sony's tendency to dump hugely expensive hardware at mass-market prices), and I have every intention of picking up more used ones over the coming months to cluster together as the networks continue to grow. Even all by it's lonesome with code that's far from optimized, the one I have is running about 10 times faster than my main desktop for roughly equivalent computations.

(Note that your mileage may vary - I just happen to like playing with systems that parallelize really well)

Re:Inexpensive, eh?

[smussman]

Yeah, but is that real money or is that subdivision expensing. In other words, does it lose $240 because Sony must use $800 of resources to produce a $600 product, or because sony-chipfab charges sony-board-assembly $60 for a part that cost $5 to produce?

I found the original analysis of the PS3 cost here [isuppli.com]. At least 50% of the components are made by other manufacturers.

Re:Inexpensive, eh?

[SailorSpork]

How much something "costs" to produce isn't as simple as the cost of parts. It may cost $5 for just the parts, it may also cost and extra $1-5 for the direct labor to put the board together, and it may cost and extra $1 to ship it. Then there's the cost of maintaining a chip fab, indirect labor and mangement costs and bribes to the chinese government to keep their cheap wage factory certified, divided over the 2-300 PS3's produced per month to keep up with worldwide demand, not to mention the cost of worldwide marketing, the cost of years of system & Blu-Ray R&D spread over each unit... keep in mind that the cost of making your PS3, Wii or software title is more than just more than the sum of the cost of the individual parts.
 
Sorry, that was my Cost Accounding class talking, I'll stop now.

Re:devaluing super

[wilsonjd]

They already do, but it is a bit more expensive than 8 PS3s: http://www-03.ibm.com/systems/bladecenter/cell-based.html [ibm.com]

Re:8 systems x 8 cores =

[this great guy]

I code on a PS3 running Linux. There are 7 cores available to Linux. 1 PPU + 6 SPU cores.
Ok the PPU is not as powerful as an SPU, it's a basic in-order dual-threaded PowerPC core with the AltiVec instruction set, but you shouldn't ignore it.

Re:Obligatory

[adisakp]

It would've been cheaper to just buy a Cray.

If you read the article and followed the link to his PS3 Gravity Grid [umassd.edu] site, you'd know a couple things about the cost (FREE) for this computational power:

#1) The total cost of purchasing an entire "PS3 Gravity Grid" supercomputer for yourself is less than the cost of a single simulation run on a BlueGene. In other words, you can buy the cow, the pasture, and a barn for the price of a gallon of milk.

#2) Sony *DONATED* his 8-node cluster (albeit with 20GB PS3's which they were closing out at the time) so he actually got a "supercomputer" for nearly free.

#3) The power of the 8-node PS3 cluster is roughly the same as a 200 node partition on a BlueGene SuperComputer (1 PS3 = 25 Blue Gene nodes). With 8 Cell CPUs, he has 56 SPU's running at ~3GHz to crank through his computations. This would mean a single CELL SPU is roughly 4X more powerful than a single BlueGene node which isn't unreasonable considering that it runs at a higher clockspeed (the supercomputer has to worry more about heat dissipation with hundreds or thousands of cores).

#4) I believe that by the US Gov't's somewhat outdated standards, a PS2 qualifies as a supercomputer. The FPU power in a PS3 is on ther order of 200 times higher than that of the PS2 for single precision and considerably more for double precision (which is emulated in software on the PS2).

Re:9 cores?

[adisakp]

If he's using the PPU core (hyperthreaded-not multicore) and 6 SPU cores, he is using 7 of 9 (1 disabled and 1 reserved of the 8 SPU's + 1 PPU) :-)

Re:Inexpensive, eh?

[budgenator]

He had one, I'm guess one he bought for personal use and developed the software on it. Once he had the software running with good performance he asked Sony for some for free, because he figured that the grant from the National Science Foundation (NSF) would be a tough sell; of course now that he has a system and its proven grants from National Science Foundation for buying PS3. The 8 PS3's give him 64 processor cores to run in parallel

Re:Obligatory

[adisakp]

Double precision floating point is emulated in software on the PS3 as well.

As a professional programmer working in the games industry (on both XBOX 360 and PS3), I can tell you that's completely untrue. You can verify this easily with information available to the general public [ibm.com] on the CELL microprocessor [wikipedia.org].

The CELL supports Double Precision in hardware. However, the SPU vector instructions only run on Single Precision which allows for up to 8 SP ops (4 X Multiply+Add's) per cycle. Double Precision is scalar (non-vectored) and also not pipelined so the throughput is much slower since DP operations can cause stalls until they complete (there are rumors that IBM is working on a CELL that pipelines DP which will help immensely). Properly pipelined and vectorized Single Precision work can be 30-50 times faster than the scalar non-pipelined DP but the CELL still has true DP hardware which is much faster than emulation by orders of magnitude.