Best Practices For Building A Multiple GPU System Tweet

[Tutor]

I. Can an Octane focused system have too many GPUs? What are the power requirements for a multiple GPU system? How to keep such a system coolest?

Evil Santa came early. He made me evict from my Tyan Server [ http://www.superbiiz.com/detail.php?nam ... 7AodjxkA_g ] my eight old GTX Titans to two Gigabyte GUP4's in SilverStone cases. Luckily, good Santa followed him. Among other things, I need your experienced water cooling and power supply tips/help. I've so far been able to acquire five GTX Titan Z Hydros at very reasonable prices. I'm looking forward to finding a few more at similar prices. I'm considering whether to install 2 Revo 350s in raid 0 on top of each of their own raid 0s or install eight Titan Z Hydros in the Tyan. If I install eight Titan Z Hydros, there will be no more empty PCIe slots. But if I install only 7 Titan Z Hydros, there will be two single wide PCIe slots available to install the Revo 350s. If I install only one more Titan Z Hydro in the Tyan ( for a total of six GTX Titan Z hydros), I'll have two additional single wide PCIe slots available to hold my GT 640/4G ( useful for interactivity while rendering) and one of my 4-port SATA extender cards to drive one of my external four port SATA drive bays. Like Nature abhors a vacuum, I abhor empty PCIe slots. Internally, the Tyan chassis has three 1000 watt PSUs (one of them acts as a backup). I have three boxed 1600 watt PSUs to help support the mod. Will one of the 1600 watt PSUs be enough of an addition to fully power 7-8 Titan Z Hydros? I'll be housing the external PSU(s) in an external frame that will also house the reservoir - http://www.swiftech.com/mcp35xreservoir.aspx , pumps [ https://www.swiftech.com/MCP35x2PUMP.aspx?variation=310 ] and radiators (rads - http://www.swiftech.com/mcrx20-xp.aspx?variation=237) since the Tyan Server wasn't designed with water cooling in mind. Each rad supports up to four 120mm fans. Will that be enough cooling? Can Octane use 13 - 16 GPUs in one system if I run more than one instance of it? Do you recommend serial or parallel water loops and can/should the two CPUs participate in a water loop with the Titan Z Hydros or should the CPUs (2xX5680s - https://www.swiftech.com/apogeelp.aspx) have their own water loop?

What say you?

Thanks in advance.

[smicha]

I'll be precise as I can:

1. Power: for each Titan Z you need 600W (Octane will draw no more than 400W). So for 6 Titans Z you need 3600W (Octane will draw 2400W), for 8 - 4800W (Octane will draw 3200W). If you have 3x1600W you can run safely 8 Titans Z, including spikes. I would connect 3 Titans Z to first 1600W PSU, next 3 Titans Z to the second PSU, next 2 Titans Z and the rest of the components to the third PSU.

2. Radiators: External rads are the must in this case. For 6 Titans Z you need about 14-16 radiator units (120 or 140mm), for 8 Titans Z - 20-24. So Aquacomouter 3360 Gigant is made for you with 24 units, each for a 140mm fan.
http://shop.aquacomputer.de/index.php?cPath=7_31_2026

3. Fans for radiator: this may sound strange but lots of guys say Noctua is the right choice - I am not one of them. Actually I am selling Noctuas - they are loud when set to 1500RPM compared to be-quiet fans or SanyDenki fans. These fans are so well built compared to Noctua

http://www2.mouser.com/ProductDetail/Sa ... z90w%3D%3D

but expensive. Currently I am testing SanyoDenki 2200RPM and it literally blows Noctua. But 2200 RPM is way too much so I am looking at 1500-1800 RPM fans to operate silently at 5V. So if you want to go with SanyoDenki 24fans x 60USD = 1440$. Then also get Lamptron fan controller for 30W per channel input.

I have 4x140 be quiet fans and - I love them - they are so quiet and efficient. I have them set at full speed all the time and I don't hear them (opposed to loud noctua at 1500RPM). So the second choice (IMO) is
http://www.bequiet.com/en/casefans/260 - you will hear silence. Literally.

But these two differ in terms of:

A. Static pressure - there is no better fan out there than SanyoDenki (5.3 mmH2O) which is a monster at 1900 RPM. Be-quiet pressure is 0.81 mmh2o at 1000RPM. So if you have hot summers get SanyoDenki and set them at low speed for regular usage.

B. BE quiet is beautifully sleeved and comes with rubber mounts. The drawback of the SanyDenki fans - you have to apply 3pin connectors on your own - they come with no box, bare cables. But you don't pay for a box and cables (as Noctua) but for the greatest quality fan.

4. Get double dual tops (EK as an example) and connect 4x D5 pumps.

5. Then use parallel connectors between water blocks. With 4 D5 pumps you'll have great water pressure to split even between 8 Titans Z. I my setup the parallel bridge gives 1C temp difference on consecutive cards.

[Tutor]

Thanks for all your wonder insights, recommendations and advice. What are you referring to here: "Get double dual tops (EK as an example) and connect 4x D5 pumps." Also, can the internal PSUs (2x1000 watts = 2000 watts total) handle the load for two of the Titan Zs and the rest of the motherboard's components (of course excluding the other Titan Zs)?

[smicha]

Thanks for all your wonder insights, recommendations and advice. What are you referring to here: "Get double dual tops (EK as an example) and connect 4x D5 pumps." Also, can the internal PSUs (2x1000 watts = 2000 watts total) handle the load for two of the Titan Zs and the rest of the motherboard's components (of course excluding the other Titan Zs)?

Dual EK top for two pumps
http://www.ekwb.com/shop/pumps-and-acce ... -pump.html

So in two such tops you'll have 4 pumps. On one of them apply reservoir
http://www.ekwb.com/shop/pumps-and-acce ... -link.html

Personally (I wish I had) bitspower tops and reservoirs so well built.



1500W is enough for 2xTitansZ and the rest of the system.

My simple rule is: 400W + no. of gpus*300 = PSU power.

[Bulwerk]

First of all, wow! this will be an epic system.

Water-cooling aside, isn't there a finite limit to how many GPUs octane can use at a given time. Isn't it 12?

[smicha]

First of all, wow! this will be an epic system.

Water-cooling aside, isn't there a finite limit to how many GPUs octane can use at a given time. Isn't it 12?

I think so. This may become an issue. I think Marcus (abstrax) could help us here.

[Seekerfinder]

First of all, wow! this will be an epic system.

Water-cooling aside, isn't there a finite limit to how many GPUs octane can use at a given time. Isn't it 12?

I think so. This may become an issue. I think Marcus (abstrax) could help us here.

12 is the current limit for local network rendering. Despite numerous attempts to get a definitive answer from Otoy regarding the amount of GPU's a single system can handle, they have not confirmed this... (unless I've missed it - ?). I think 8 GPU's is the most I saw anyone here use and at least one demo from Otoy shows 8 GPU's in action.

There is a guy on the Nvidia forums that got something like 12 or even 16 running on a backplane. But I don't think he was running Octane. I think it was a Linux system (?)

I'd really like to know what determines the limit - Octane or the OS. And it would be great to finally get a definitive answer from Otoy.

Seeker

[rappet]

...12 is the current limit for local network rendering. Despite numerous attempts to get a definitive answer from Otoy regarding the amount of GPU's a single system can handle, they have not confirmed this... (unless I've missed it - ?). I think 8 GPU's is the most I saw anyone here use and at least one demo from Otoy shows 8 GPU's in action. ...
Seeker

I can confirm using 11 gpu rendering one Octane scene, with 4GPU on the master and 3 on one slave and 4 on an other slave.
Cheers,

[p3taoctane]

Mu understanding is that Windows only supports 8 total in your box
So 4 Z's is the limit in one box under that OS

You may be able to fit 8 watercooled Z's but do check the actual backplate of the card as it takes up three slots even though your water cooled dimensions are thinner. Unless you trim down the bracket or use a riser you will run into space.

I may be wrong in that the water cooled unit comes with a new attachement plate... I'm not sure on that

Sounds like a great box though.

[smicha]

Mu understanding is that Windows only supports 8 total in your box
So 4 Z's is the limit in one box under that OS

You may be able to fit 8 watercooled Z's but do check the actual backplate of the card as it takes up three slots even though your water cooled dimensions are thinner. Unless you trim down the bracket or use a riser you will run into space.

I may be wrong in that the water cooled unit comes with a new attachement plate... I'm not sure on that

Sounds like a great box though.

EK waterblocks for Titan Z comes with a dual slot bracket that replaces a triple bracket
http://www.ekwb.com/shop/blocks/vga-blo ... ickel.html

My question to OTOY - if we have 8x Titans Z, and if the limit on one board is 12gbus, will extra 4 gpus be seen i Octane via slave licence. Shortly speaking - what is the max limit of gpus on regular+slave licence and how the gpus have to be distributed?