Best Practices For Building A Multiple GPU System

[glimpse]

Pure madness =) but so is all wasted space..

I mean having single system that is so powerfull is just insane,
But so is the scale & the mess ..do not get me wrong, but it's like..
american madness with cars - bigger is better,
but is it the case here with rendering systems?

I hope this all is just a "sketch" & You'll polish those, Tutor =)
Merry X-mas, fellow geeks! =)

[smicha]

Merry Christmas to all of GPU enthusiasts

[Notiusweb]

Here's that update that I promised to give:
System no. 1 Goal = Have four PSUs, motherboard, lots of storage, et., and 20+ water touched GPU processors all in one case.

Hi Tutor, looks great. Thanks for the pics! I myself found that in the mid stages of building, it's almost better to have things very accessible and not try to commit to one look. Your setup sounds like it will be in optimization flux as you add in, and the flux is okay. It involves a lot of trial and evolution. You know, at times I was terrified of my rig, but then after a while it all seems completely tame and unified. The watercooling part is very rewarding in my opinion, worth every bit of flux encountered when setting up. Make sure you 'Titan-Z' screws good!

Itou31 and myself have an interesting rig similarity; we both have watercooled multi-gpu rigged pc's that serve as dual mode:
(1) Primary mode - a single primary GPU is active and all the external GPUs are off, and
(2) Render mode - a full multi-GPU mode, where the primary and all of the external GPUs are active.

This is made possible by having the main GPU and Mobo running off one PSU, and then all the other stuff on separate external PSUs, which themselves are not connected to the mobo, but rather are switched on prior to booting up.
I am curious, is your unit all or none, or can you, at your discretion, deactivate PSUs to achieve different GPU configs? I have found it a rather 'nifty' feature to be able to operate the PC as a standard desktop when designing, and then activate the whole shebang when rendering. It gives the option of not having to run the system full-Monty for every little thing. Cool idea to toy with when building, just a matter of preference, ultimately. For me, it makes me want to put the strongest GPU as primary.

Anyways, great work, excited to see more!

PS - look at this. He does it for coin mining, but really interesting presentation of his rig.
https://www.youtube.com/watch?v=Wh7TFqgddCk

[Tutor]

Merry Christmas to all of GPU enthusiasts

Happy New Year to all.

Smicha,

Thanks for words of wisdom and encouragement in 2014 and 2015 and I pray that the New Year rewards you and your family greatly.

[Tutor]

Notiusweb,

Thanks for your words of wisdom and encouragement in 2015 and I pray that the New Year rewards you and your family greatly.

... .

Hi Tutor, looks great. Thanks for the pics! I myself found that in the mid stages of building, it's almost better to have things very accessible and not try to commit to one look. Your setup sounds like it will be in optimization flux as you add in, and the flux is okay. It involves a lot of trial and evolution. You know, at times I was terrified of my rig, but then after a while it all seems completely tame and unified. The watercooling part is very rewarding in my opinion, worth every bit of flux encountered when setting up. Make sure you 'Titan-Z' screws good!

Itou31 and myself have an interesting rig similarity; we both have watercooled multi-gpu rigged pc's that serve as dual mode:
(1) Primary mode - a single primary GPU is active and all the external GPUs are off, and
(2) Render mode - a full multi-GPU mode, where the primary and all of the external GPUs are active.

This is made possible by having the main GPU and Mobo running off one PSU, and then all the other stuff on separate external PSUs, which themselves are not connected to the mobo, but rather are switched on prior to booting up.
I am curious, is your unit all or none, or can you, at your discretion, deactivate PSUs to achieve different GPU configs? I have found it a rather 'nifty' feature to be able to operate the PC as a standard desktop when designing, and then activate the whole shebang when rendering. It gives the option of not having to run the system full-Monty for every little thing. Cool idea to toy with when building, just a matter of preference, ultimately. For me, it makes me want to put the strongest GPU as primary.

Anyways, great work, excited to see more!

Here're just a couple of examples of the importance of your contributions to which I refer earlier in this post:

I originally intended to have three of seven GTX Titans on the motherboard, running the other GPUs off of splitters and/or riser cables, and connecting those GPUs to the upper rail. However, this particular post of yours reminded of earlier posts of yours where you spoke about the limits to overclocking lots of GPUs given current software tweaking limitations (in AfterBurner as I recall) regarding accessing all of the many GPUs. Since I'll be running three different GPU groups (1 - 6xTitanZs, 2 - 7xTitans and 3 - 1xTitan Black), your earlier posted experience has motivated me to install the Titan Black, one of the Titans - the one fully water-cooled, and one of the Titan Zs on the motherboard instead. That way, I hope to improve my chances of getting Afterburner to allow me to see and tweak the Titan Black individually, the TitanZs, at least, as a group and the Titans, at least, as a group. I'm now in the process of testing the GPUs for optimal group tweaking.

Almost all of my projects involve AV production (these are also GPU accelerated tasks) and rendering animation (and my using multiple GPU accelerated 3d rendering and video effects renderers simultaneously). Therefore, given what I do I need multiple GPUs to be active a bit earlier in the process for this system. But I can power the Titan Black (slotted on the motherboard), the Titan (slotted on the motherboard) and the Titan Z (slotted on the motherboard) [ 350W + 350W + 550W ] = 1,250W*/ ] from the 1,600W LEPA PSU powering the motherboard. When necessary, I can run the five Titan Zs and six Titans (550W * 5 + 350W * 6 = 4,850W*/) on the upper rail, powering them with the three 1,600W EVGAs and the 450W BoosterX5 (5,250Ws total capacity w/o counting PSU peaking ability). At those times when I do not need the lots of GPU rendering resources, I'll not flick the"ON" switch on the three 1600W EVGA PSUs and simply unplug the BoosterX.


*/ In this and in my earlier post on the same subject, I use(d) these inflated numbers to provide maximum headroom for tweaking GPU rendering speeds.

PS - look at this. He does it for coin mining, but really interesting presentation of his rig.
https://www.youtube.com/watch?v=Wh7TFqgddCk

Many thanks for the YouTube reference. Since Jerry (the author) and I aren't far apart (he resides in Alabama also), I'll soon try to reach out to him to learn a bit more about coin mining and mining rigs and mining technology and how it all may related to 3d rendering rigs and to determine the other benefits.

[Tutor]

Pure madness =) but so is all wasted space..

I mean having single system that is so powerfull is just insane,
But so is the scale & the mess ..do not get me wrong, but it's like..
american madness with cars - bigger is better,
but is it the case here with rendering systems?

I hope this all is just a "sketch" & You'll polish those, Tutor =)
Merry X-mas, fellow geeks! =)

My car is a 2000 Toyota Celica GT-S. Many of my neighbors have small cars that were made by Japanese, German and other "foreign" car companies. And, "yes," some have cars made by American car companies, but most of those cars are small cars. Thus, the brush with which you paint is too wide and too old.

While to you it may appear to be pure, insane madness, obviously I'm not building systems for your needs and this is one of them. I'm building systems to meet my particular needs. Until someone invents and implements safe, wireless electrical transmission, much smaller GPUs, and reasonably priced multi-GPU oriented systems, I'll be traveling the same road that I'm on now, "mess" and all. My systems are for getting my work done; they're not the work itself. "Yes, I'll polish it." But where I'll concentrate the polishing will be on system performance. I can't make the cables disappear and still achieve the level performance that I'm after.

P.S. I'm, however, looking forward to seeing that 1,000 point OctaneBench score posted for 7 x GTX780 6Gs - now that score would surprise me.

[Notiusweb]

Look at this! Love seeing this! I don't know how long this was there...

From Octane Bench:

12x Tesla K80 1 result

Maximum 876.44 Average 876.44
Minimum 876.44 Median 876.44

This card has:

Memory size (GDDR5) : 24GB (12GB per GPU)
CUDA cores: 4992 ( 2496 per GPU)
Memory bandwidth: 480 GB/sec (240 GB/sec per GPU)
2.91 Tflops double precision performance with NVIDIA GPU Boost

http://www.amazon.com/Nvidia-Accelerato ... B00Q7O7PQA

Maybe they used this - CA16000- it can power 16 GPUs at PCI X16

"Four removable canisters house up to four full-height, full-length, PCIe x16 double-wide GPUs each. The system is powered by three 3000-watt redundant power supplies and includes an IPMI-based system monitor."

http://www.onestopsystems.com/3u-comput ... tesla-gpus

[glimpse]

Notius, 12 k80s means 6 GPU cards & result is pathetic considering the price You have to spend for those =) you can not justify PRO cards for octaneRender..at all.. That result could be reached for fracture of price. Even with 12GB equiped TitansXs - 6 of them watercooled would cost no more than 10k, while the system You pointed with GPUs You mentioned would stand at least three/four times more expensive =)

Tutor, there's no way to reach 1000 in OB using 7x GK 110 units(if You care about stability)but it's possible using GM 200. Regarding to the same topic, EK waterblocks is making 7 port bridge for single slot waterblock equiped GPUs =) this will allow easy way to plug seven cards without need of EOLed fittings in between..

Here is my last build in progress, eGPU box giving 400 in terms of Octane Bench (OCed using Afterburners, in Windows) with 4x 780s & two 240 rads in mATX case (with GM 200 chip based GPUs like 980Ti or TitanXs result would be 600 easily).

I'll write article about this route, in's & out's, but from all I learned so far, this seems to be one of the most logical options I see right now.

image.jpg

[Notiusweb]

Postby glimpse » Thu Dec 31, 2015 8:51 am
Notius, 12 k80s means 6 GPU cards & result is pathetic considering the price You have to spend for those =) you can not justify PRO cards for octaneRender..at all.. That result could be reached for fracture of price. Even with 12GB equiped TitansXs - 6 of them watercooled would cost no more than 10k, while the system You pointed with GPUs You mentioned would stand at least three/four times more expensive =)

It's interesting, I don't have any info as to how that rig came to be, or what other uses those cards may have, but seeing the "12" all by itself spins my mind. I wonder what arrangement was used to hit that number, and how far beyond that number they could go, etc. I would certainly invite that party to partake in this forum and share what goals they have, or how rendering plays into their schematic of GPU usage. Even if I hit the higher score for 12, theirs would double the memory of my scenes!

Here is my last build in progress

Cool! Wow, looks so compact. Was this your first multi-GPU build? Excited to hear more, great work, very Smicha-esque!
How many GPUs could your rig support if you fully embraced the dark side of the 'Board'? Do you set one aside as primary display, and use the other 3 to render, or would you go full on with the 4 usually. Is one, or all of them , Titan Blacks?
Also, if you can, add external ones with chords all over the place. Even if you can't add yet, just take them out and make it external for the hell of it Look forward to the article.

[smicha]

As far as I know this is a custom made gpu expansion box based on an Amfeltec board, isn't it Tom?

Tom, if I may ask - how much is the Amfeltec board, and have you got it from Canada? How does the system connect with a PC or Mac - does it require a free PCIe slot on a master computer? Any additional software from Amfeltec?

As for the discussion on various chips performance and Octane score - no matter what chip is used every single Cuda core adds roughly about 1/23 (to 1/21) to Octane score, i.e., Old Titan 2688/23 = 116, Titan X 3072/23=133 which is close to real benchmarks. The limit here is just core clock and (in 1/4) memory clock.

Love this topic, Tutor. Happy new 2016 to all of you, guys and I hope Pascal will bring us some fresh air.