Best Practices For Building A Multiple GPU System

[Tutor]

I wish all of you a happy and prosperous 2016 (and afterwards).

Look at this! Love seeing this! I don't know how long this was there...

From Octane Bench:

12x Tesla K80 1 result

Maximum 876.44 Average 876.44
Minimum 876.44 Median 876.44

This card has:

Memory size (GDDR5) : 24GB (12GB per GPU)
CUDA cores: 4992 ( 2496 per GPU)
Memory bandwidth: 480 GB/sec (240 GB/sec per GPU)
2.91 Tflops double precision performance with NVIDIA GPU Boost

http://www.amazon.com/Nvidia-Accelerato ... B00Q7O7PQA

Maybe they used this - CA16000- it can power 16 GPUs at PCI X16

"Four removable canisters house up to four full-height, full-length, PCIe x16 double-wide GPUs each. The system is powered by three 3000-watt redundant power supplies and includes an IPMI-based system monitor."

http://www.onestopsystems.com/3u-comput ... tesla-gpus

Once upon a time not to long ago I was transported to a galaxy far-far away, and this discussion, whimsically started by me, ensued: viewtopic.php?f=40&t=43597&start=200#p242677 . Then after investigating the prices/performance, I ended up back on earth here: viewtopic.php?f=40&t=43597&start=210#p242697 and where I stand today.

Notius, 12 k80s means 6 GPU cards & result is pathetic considering the price You have to spend for those =) you can not justify PRO cards for octaneRender..at all.. That result could be reached for fracture of price. Even with 12GB equiped TitansXs - 6 of them watercooled would cost no more than 10k, while the system You pointed with GPUs You mentioned would stand at least three/four times more expensive =)

Tutor, there's no way to reach 1000 in OB using 7x GK 110 units(if You care about stability)but it's possible using GM 200. Regarding to the same topic, EK waterblocks is making 7 port bridge for single slot waterblock equiped GPUs =) this will allow easy way to plug seven cards without need of EOLed fittings in between..

Here is my last build in progress, eGPU box giving 400 in terms of Octane Bench (OCed using Afterburners, in Windows) with 4x 780s & two 240 rads in mATX case (with GM 200 chip based GPUs like 980Ti or TitanXs result would be 600 easily).

I'll write article about this route, in's & out's, but from all I learned so far, this seems to be one of the most logical options I see right now.

image.jpg

There May Come A Point In The Ouch Realm When Self-Reliance Kicks In

I agree with all of your above quoted points. That build that you have underway looks great. Please keep us apprised concerning it's progress, as it looks like an excellent build for you and others specialising in artificial photography (visualisation) for Architecture, Design and Development. We, as Octane users, have a great diversity of needs for Octane helps support a very broad family with a myriad of skills and needs. As to that One-Stop System box, referenced by Notiusweb, which holds up to 16 GPUs, the last time that I looked at it with any interest, it cost [w/4 (at a minimum/mandatorily included) GTX Titan Xs] - $24,750 and went up to ~ $39,800*/ for 16 Titan Xs. It was, then, one of the lowest priced options out there that I could find for a ready-made massive GPU housing system. However, it's headless, i.e., one still needs a computer to connect it to (Ouch!) and the One-Stop Systems told me that they, at least then, wouldn't sell it to me with less than four pricey GPUs - Double Ouch! If one were to note my next posts following that I'm sure that they could glean that I decided to continue going another route - self-builds. I know they will not be beautiful like your custom build is or like a One-Stop-Shop box would be. I'm sure that you chose a build that suits your business' needs. I run a small business providing corporate training programs (it includes producing 3d animations and AVs) and my costs have to be controlled tightly. If you have some ideas about how I could accomplish my goals and meet my needs for lots of GPUs in as few boxes as possible and at the least possible costs, and still polish them so they would be beautiful like your box, I'm always open to hearing constructive, specific suggestions. I started this journey because I had reached 24 systems and the workspace, power, cooling, management and software licensing costs were eating at my profits like wolves on a lamb. Running my many GPUs in as few boxes as possible is one way I see to cut and contain those costs. My clients never see my physical rendering network and care only about the final production - so the way the systems look is immaterial from a profit standpoint. So for now I'll just build my own systems and live with their bare-utilitarianism and whatever valuable input others like you care to provide that's consistent with my goals.


*/ Evidently, One-Stop System was, at least then, valuing an air cooled base Titan X at a sales price of over $1,250.00 (USD) [ $39,800 - $24,750 (that includes 4 Titan Xs) / 12 = $1,254.17 ]. But maybe that $200+ margin was just to have those other GPUs personally pre-inserted. Mini-Ouch! - a third to the fourteenth time - for the insertion of each of the other twelve GPU cards.

[Notiusweb]

There's an old Amish saying:

"Use as many GPUs and PSUs as you G*d D@mn can..."

However, in the case of the Tesla K80, it's only part of what happened.

Tesla K80.jpg

The scores, to me, show a pattern of benchmarking for specific use levels. I don't think users typically bench their GPUs this way, which leads me to believe this may be some sort of test run, or data logging, for other use than rendering. Perhaps the manufacturer of a unit holding the GPUs, quality controlling for some system to be delivered??

In any event they have the capacity to hold 12 GPUs, and to run an OS with 12 GPUs. And with what Tutor said...

Evidently, One-Stop System was, at least then, valuing an air cooled base Titan X at a sales price of over $1,250.00 (USD) [ $39,800 - $24,750 (that includes 4 Titan Xs) / 12 = $1,254.17 ]. But maybe that $200+ margin was just to have those other GPUs personally pre-inserted. Mini-Ouch! - a third to the fourteenth time - for the insertion of each of the other twelve GPU cards.

...it is only a matter of time before Titan X package is realized, and with that, it could be up to 16 GPUs.

Thus finally, may I dare say:

A 12x score for Titan X is coming....

[Tutor]

There's an old Amish saying:

"Use as many GPUs and PSUs as you G*d D@mn can..."

However, in the case of the Tesla K80, it's only part of what happened.

Tesla K80.jpg

The scores, to me, show a pattern of benchmarking for specific use levels. I don't think users typically bench their GPUs this way, which leads me to believe this may be some sort of test run, or data logging, for other use than rendering. Perhaps the manufacturer of a unit holding the GPUs, quality controlling for some system to be delivered??

In any event they have the capacity to hold 12 GPUs, and to run an OS with 12 GPUs. And with what Tutor said...

Evidently, One-Stop System was, at least then, valuing an air cooled base Titan X at a sales price of over $1,250.00 (USD) [ $39,800 - $24,750 (that includes 4 Titan Xs) / 12 = $1,254.17 ]. But maybe that $200+ margin was just to have those other GPUs personally pre-inserted. Mini-Ouch! - a third to the fourteenth time - for the insertion of each of the other twelve GPU cards.

...it is only a matter of time before Titan X package is realized, and with that, it could be up to 16 GPUs.

Thus finally, may I dare say:

A 12x score for Titan X is coming....

One-stop systems’ latest phenom is the GPUltima which holds 128 GPUs - http://www.onestopsystems.com/gpultima . “The GPUltima is a single 19" rack comprised of 8 OSS High Density Compute Accelerators (HDCA) each with 16 NIVIDA Dual GPUs (128 total*/), 16 dual-socket servers, an Infiniband Switch and an Ethernet Switch.” But, given the price for a single 16 GPU chassis [ http://www.onestopsystems.com/4u-comput ... tx-titan-x ], the GPUltima’s price, about which I haven’t had the nerve or interest to ask, would likely trigger, at least in me, terminal “Ouch.” One-Stop Systems’ focus appears to be on the Tesla market, which we 3d artist aren’t usually interested in because Teslas do not perform as well with current 3d software as do the GTXs [ See, e.g., example - https://render.otoy.com/octanebench/res ... ingleGPU=0 . I agree that the company could be testing their Tesla systems since those are the products One-Stop sells that appear to hold the greatest potential for moneymaking for the company. Thus, subsequently the company could do the same for their maximally overpriced Titan systems and thus a 12x score for Titan X could come sooner thanks to One-Stop Systems. However, I’m not focused on double-precision floating-point tasks in my 3d/AV work (although I'm in the process of learning CUDA coding in all of its facets) and thus if One-Stop were testing one of its systems with Teslas, in addition to my questioning the wisdom of their pricing for a Titan X system (which such floating-point users would likely pass up for a Titan-Z or even the original GTX Titan [both of which have vastly superior double-precision floating-point prowess over any Maxwell]) or just wait for Pascals and Voltas which are said will be the ultimate in floating point monsters, I’d also be questioning One-Stop’s wisdom in using Octane to bench Tesla performance. There are much better tests for higher precision floating-point prowess. Most knowledgeable Tesla owners probably don't care about the measurements Octane Bench yields, although Octane (and other 3d app) users (or potential users) might be lead to make better purchasing decisions by being armed with the knowledge that for their use GTXs are better purchases.

*/ I admit that I could use the GPUltima fully loaded if it were priced within the realm of reasonableness, to house my 108 CUDA GPU rendering processors (excluding my 7 GT 640 4Gs) and 12 hulking OpenCL-only processors [Thus, I currently have 127 compute processors counting the 7 GT 640 4Gs and anticipate having even more when we get the Pascals beginning this year, and in 2018 begin getting the Voltas). Very likely, my Amish distant cousins have long been shouting to me in my sleep - "Use as many GPUs and PSUs as you G*d D@mn can...".

[Notiusweb]

LOL

the GPUltima’s price, about which I haven’t had the nerve or interest to ask, would likely trigger, at least in me, terminal “Ouch."

And you wrapped it up nicely by boomeranging the quote at the end

[smicha]

https://www.youtube.com/watch?v=LXOaCkb ... ploademail

7 WC GPUs with the case I keep telling is the best. If I had a choice I'd exchange my SMH10 with S8.

[Tutor]

https://www.youtube.com/watch?v=LXOaCkbt4lI&feature=em-uploademail

7 WC GPUs with the case I keep telling is the best. If I had a choice I'd exchange my SMH10 with S8.

Mindblowing rig!

[Notiusweb]

Re: Best Practices For Building A Multiple GPU System
Postby smicha » Sun Jan 03, 2016 9:07 am
https://www.youtube.com/watch?v=LXOaCkb ... ploademail

7 WC GPUs with the case I keep telling is the best. If I had a choice I'd exchange my SMH10 with S8.
Watercooled TITAN+3x780_6GB@1300mHz/7000mHz, [email protected], P8P67WS, 32GB, 256GB, 2x2TB WD EARX, 1350W
CASE-LABS SMH10, EK, Airplex_480, SR1_560, 20xNB_PL2, 2xD5, Aquaero 6XT
build-log http://render.otoy.com/forum/viewtopic.php?f=9&t=42540

He could add a 2nd 1600 PSU with idea of running it through an extension cord to another breaker, and have a rendering beast as well with 7 Titan Xs.

Simcha, S8 to add more GPUs?

[smicha]

S8 'cause it's half of the size of STH10 and can handle 3x360 rads easily. Plus a pedestal is available.

[Notiusweb]

Smicha, might you add a rad outside the case in the pedestal? Why let the case stop you .
Or does the case itself mark the boundary, as it were, for adding components.

[smicha]

2x 360 in the pedestal and pedestal over a pedestal is possible