Best Practices For Building A Multiple GPU System Tweet

[Tutor]

I'll be precise as I can:

1. Power: for each Titan Z you need 600W (Octane will draw no more than 400W). So for 6 Titans Z you need 3600W (Octane will draw 2400W), for 8 - 4800W (Octane will draw 3200W). If you have 3x1600W you can run safely 8 Titans Z, including spikes. I would connect 3 Titans Z to first 1600W PSU, next 3 Titans Z to the second PSU, next 2 Titans Z and the rest of the components to the third PSU.

2. Radiators: External rads are the must in this case. For 6 Titans Z you need about 14-16 radiator units (120 or 140mm), for 8 Titans Z - 20-24. So Aquacomouter 3360 Gigant is made for you with 24 units, each for a 140mm fan.
http://shop.aquacomputer.de/index.php?cPath=7_31_2026

3. Fans for radiator: this may sound strange but lots of guys say Noctua is the right choice - I am not one of them. Actually I am selling Noctuas - they are loud when set to 1500RPM compared to be-quiet fans or SanyDenki fans. These fans are so well built compared to Noctua

http://www2.mouser.com/ProductDetail/Sa ... z90w%3D%3D

but expensive. Currently I am testing SanyoDenki 2200RPM and it literally blows Noctua. But 2200 RPM is way too much so I am looking at 1500-1800 RPM fans to operate silently at 5V. So if you want to go with SanyoDenki 24fans x 60USD = 1440$. Then also get Lamptron fan controller for 30W per channel input.

I have 4x140 be quiet fans and - I love them - they are so quiet and efficient. I have them set at full speed all the time and I don't hear them (opposed to loud noctua at 1500RPM). So the second choice (IMO) is
http://www.bequiet.com/en/casefans/260 - you will hear silence. Literally.

But these two differ in terms of:

A. Static pressure - there is no better fan out there than SanyoDenki (5.3 mmH2O) which is a monster at 1900 RPM. Be-quiet pressure is 0.81 mmh2o at 1000RPM. So if you have hot summers get SanyoDenki and set them at low speed for regular usage.

B. BE quiet is beautifully sleeved and comes with rubber mounts. The drawback of the SanyDenki fans - you have to apply 3pin connectors on your own - they come with no box, bare cables. But you don't pay for a box and cables (as Noctua) but for the greatest quality fan.

4. Get double dual tops (EK as an example) and connect 4x D5 pumps.

5. Then use parallel connectors between water blocks. With 4 D5 pumps you'll have great water pressure to split even between 8 Titans Z. I my setup the parallel bridge gives 1C temp difference on consecutive cards.

Thanks Smicha for all of that most useful advice. You've really got me to think more deeply about my computer system needs and priorities. Your advice/formula for power requirements has already led me wisely forgo purchasing a 7th or 8th Titan Z, and thus also make way for serving my 4k video productionsneeds with the Revo 350s, GT640 and SATA drive backup. That Aquacomputer 3360 Gigant is a monster of an external rad. I'm sure that it's a magnificent cooling component. Your pointing out the various water cooling consideration for my computer made me also think about serving my human water cooling requirements. I'm considering purchasing a small refrigerator for this computer room. Of the two refrigerators that I'm considering, each is extraordinarily quiet, consumes just 83 watts and can be adjusted within the temperature range of 36ºF - 56ºF temperature range. Their only significant difference is size and cost. Here are fuller specs:

(1) Avanti 1.7 Cu. Ft. Superconductor Refrigerator - Stainless Steel
Model:SHP1702SS
Price:$89.97 US
Brand: Avanti Solid state components for reliability
No vibration
Adjustable thermostat
Auto Defrost
36ºF - 56ºF temperature range
Environmentally friendly
Soft interior LED light that operates when the door is open
Reversible door swing
Tall bottle door rack - holds a 2 liter bottle
Slide-out shelf with three positions
Shipping weight: 31 pounds; net weight: 27 pounds
No compressor step- no compressor!
Power: 83 watts, 115V, 1.1 amps
Warranty: One year parts and labor
Dimensions: 20.5" H x 17" W x 18.75" D

or

(2) Avanti 2.5 Cu. Ft. Superconductor Refrigerator - Black
Model: SHP2501
Price:$125.97 US
Brand: Avanti Solid state components for reliability
No vibration
Adjustable thermostat
Auto Defrost
36ºF - 56ºF temperature range
Environmentally friendly
Soft interior LED light that operates when the door is open
Reversible door swing
Tall bottle door rack - holds a 2 liter bottle
Slide-out shelf with three positions
Power: 83 watts, 115V, 1.1 amps
No compressor step
Shipping weight: 45 pounds; net weight: 40 pounds
Warranty: One year parts and labor
Dimensions: 29 1/2" H x 17" W x 20 1/2" D

I can also set the reservoir on top of either of them to increase liquid/head pressure to my computer system, which will I'll set closer to the floor.

Do you recommend PrimoChill or another brand of non-conductive coolant or are there other coolant related considerations that I should take into account? Should I also use separate leak detection and temperature and pressure sensors and, if so, what brand(s) and model(s) do you recommend?

[smicha]

Coolant - I use distilled demineralized water - 5 liter for 1EUR. Works fine. Some time ago I used expensive UV coolants. I don't see a need for it anymore. You may add some concentrate, mayhems.

Sensor - one per loop in the hottest spot, e.g. next to a radiator inlet.

[Tutor]

Coolant - I use distilled demineralized water - 5 liter for 1EUR. Works fine. Some time ago I used expensive UV coolants. I don't see a need for it anymore. You may add some concentrate, mayhems.

Sensor - one per loop in the hottest spot, e.g. next to a radiator inlet.

Smicha

Since this is my first try at building a water cooling system from scratch, I'll play it cautiously and use a non-conductive liquid. As my skills improve in your direction, I may not feel the need to be so cautious.

Please give me your opinion about my immediately following Mr. Freeze. What am I missing?

Also, I plan to provide each Titan Z with up to 750 watts to play with. Also, I picked up a Titan Black Hydro Copper Signature Ed. which will have up to 650 watts to play with. Non-gpu components will have 500 watts to play with. Do you see any problems or pitfalls so far? Thanks.

[glimpse]

Tutor, I don't think refrigerator is going to cope with the heat =) they do tend to keep temp pretty stable,
but it's too much heat even from couple of GPU's for fridge =) I might be wrong though, do Your homework =)

there are such cooling box'es (check phase cooling) with CPU waterblocks but they are not up to levels to keep GPUs cool =)
so I would skip this an step into sort of typical waterCooling loop with good external rad =) it makes most sense, pluss..
if You get enough capacity, You could go pretty low in terms of noise & temps =) pluss it's more reusable =)

[Tutor]

Well, only sort of. I know that I shouldn't let the coolant freeze because then it won't flow properly and thus can't do it's job and might cause leaks etc.. But how about keeping the coolant's temperature above it's freezing point, say - within a range of at about 5-13 degrees celsius or 40-55 degrees Fahrenheit? I believe that I will be able to boost the freezer's cooling capacity when the GPUs are under load to keep them at about 55 degrees Fahrenheit (12.78 degrees Celsius) and otherwise maintain them at about 40 degrees Fahrenheit (4.44 degrees Celsius) when the GPU's aren't under load.

Tutor, I don't think refrigerator is going to cope with the heat =) they do tend to keep temp pretty stable,
but it's too much heat even from couple of GPU's for fridge =) I might be wrong though, do Your homework =)

there are such cooling box'es (check phase cooling) with CPU waterblocks but they are not up to levels to keep GPUs cool =)
so I would skip this an step into sort of typical waterCooling loop with good external rad =) it makes most sense, pluss..
if You get enough capacity, You could go pretty low in terms of noise & temps =) pluss it's more reusable =)

Thanks Glimpse for your insights. Now I've found an alternative - Mr. Freeze(r).
I intend to use a refrigerature-freezer ("freezer"), whose temperature I can control precisely and lower greatly, in combination with a typical water cooling loop. Two of the four external radiators' fans will be drawing upon ambient air to cool the liquid (just before returning it to the reservoir/pump) and the other two of those external radiators will be in the freezer doing the same thing (to the coolant right after it has left the reservoir/pump), but imposing much cooler refrigerated air, on the same loop. The freezer's inside temperature will be a lot cooler than the ambient air in the room where the system will be located. The temperatures of the two radiators exposed solely to ambient air (depending on placement) should, because of that alone, be a little cooler than radiator(s) located completely inside a computer and being subject to the internal temperatures there. I'll use two dehumidifiers to suck the ambient moistness out of the air, to help further to cope with the effects of the differences in temperatures. The Avanti refrigerator/freezer model [ avanti-vfr14ps~7AVNI031.htm ] that I finally ordered costs ~ $105 (US) and can be set as low as 0 degress Fahrenheit or -17.78 degrees Celsius. The temperature of that whole unit can also be set to make it entirely behave as a freezer. I'm thinking about having two coolant temperature sensors - (1) one sensor pre-entry to the two radiators (connected serially) cooling the coolant ( that has cooled the components ) that will next go to the reservoir/pump and (2) one sensor after the coolant has left the freezer headed to cooling the components. That way, I can also keep track of deltas, knowing (1) the rise in temperature from cooling the components, (2) the coolant's temperature at that stage, (3) how much the external, ambient air cooled radiators, combined with the freezer (and the two radiators in the freezer) have cooled the coolant and the temperature of the coolant being sent to the components . Thus, I should be able to set the freezer's temperature more appropriately as time goes on. I do intend to try this first without having fans on the radiators within the refrigerator; but i may find that I need special fans to operate properly/safely at low temperatures or that by using fans, I do not have to set the freezer's thermostat as low.

I intend to have my reservoir/pump sitting high about the freezer to increase pressure.

If using the Avanti doesn't work out as I intend, I'll just patch the Avanti and dedicate it solely to the manufacturer's likely sole intended use, i.e., just for food storage. Then all four radiators, with all of their fans, will be cooled solely by ambient air (outside of the subject computer's case, of course).

Do you believe that Mr. Freeze(r) will not better help to cope with the heat than just using a traditional water loop alone where all four radiators would be just relying on ambient air alone to cool the liquid. Am I missing something? Thanks Glimpse.

P.S. Giving cold credit where it is due, I got this unconventional lead from here - http://www.overclockers.com/extreme-wat ... igeration/ , but in my mod, the only thing of significance that will be in the freezer ( excluding bottled drinks for me - in aid of this chilling effect maybe I'll let the drinks freeze in their plastic container after I've dranked enough from them to allow for freezing expansion) will be the two radiators (without or with fans [if required to realize my aims]). Also, I intend to put only one hole on each side of my freezer - one to let the air cooled coolant ( that has gone through the reservoir/pump in and the other to let the chilled coolant out. Moreover, I don't mind having to adjust freezer settings before powering up, shutting down, or as needed on the fly.

[Tutor]

Here're some pics of what will be the guts of Mr. Freeze(r) and his assistants.

1) First Pre-Chilling: My two pre-chillers (Mr. Freezer's 18 fan assistants) are shown in pics 1 & 2 (Two Watercool MO-RA3 Pro 9x120mm/4x180mm Extreme Radiators). Each MO-RA3 uses 28 m copper tubing and 2.5 square meter of aluminium fins for heat transfer from cooling fluid to ambient air. When going through each radiator the cooling fluid passes through 72 tubes in a four-time parallel layout. Coolant will flow through these two radiators serially, i.e., these two radiators will be placed side by side.

2) The In Freezer Chillers: Pics 4 & 5 show the chilling units outside of the mini the freezer (see Pic 3) where they will reside. Those pics show the relative sizes of (a) the Alphacool NexXxoS Monstra Dual 120 mm Radiator ([the thick (80mm) stage 1 chiller] on top of the box - it's goes into the top section of the Mr. Freeze(r) to be feed first from the two serially connected MO-RA3 radiators; then, it feeds the coolant to (b) four Aquacomputer Airplex Modularity System 240 Radiators (each with 21 copper pipes ) in front of the box (stage 2 or main chillers). Those four radiators will sit in the front/bottom of Mr. Freezer.

3) The Chillers in Place: In the third pic (the installed view) are 4 ganged together Aquacomputer Airplex Modularity System 240 Radiators with four fans. Behind them in the upper part of Mr. Freeze(r) is the Alphacool NexXxoS Monstra Dual 120 mm Radiator (the stage 1 chiller with two fans). Total fan count is 24, but because of how the four System 240 Radiators are ganged together, there are actually 28 120mm surfaces that will be cooled by 24 fans and next passed to my reservoir and pumps. The reservoir and pumps will be the highest placed components to increase head pressure in the entire coolant loop.

4) I will use two low wattage dehumidifiers to control room humidity external to Mr. Freeze(r). I will use silicon granules and jells to control humidity/condensation inside Mr. Freezer until my Rosahl Solid Polymer Electrolytic MDL-7 Dehumidifier arrives from across the pond.

5) After further researching the obstacles I'm facing, I still intend to reach my goal of keeping the GPUs under 56 degrees Fahrenheit (12.78 degrees Celsius) under load and otherwise maintain them at about 40 degrees Fahrenheit (4.44 degrees Celsius) when the GPU's aren't under load or are idle.

6) I will tackle other parts of this mod as parts arrive and hope to finish this project before Xmas. Again, thanks to all of the forum members who have participated in this thread, for your participation has helped me to better understand all of the various considerations I needed to process for this project to succeed.

[glimpse]

Hi, Tutor!

Your radiator pick is awesome! MoRa 9x120mm + aquacomputer 240mm is nice step.

So, You going to end up cooling 4x TitanZs or more? =)

Even for 4x TitanZs these two MoRas should be enough to keep the temperature 10-15 C higher then ambient.

Now I migt be wrong, but those rads in fridges are not going to help, but make that fridge act like an oven I want to be wrong , but I migh be right I don't believe thei are going to cope with heat from rad. Especially keeping in mind how much air is in them

What is the ambient temperature of Your room going to be? Hope it's pretty low - the best case scenario i believe You can get is no lower temps than ambient (if those freezers are not going to fail)..but hey it's nice experiment & I hope I'm wrong

Really curious how it's going to unfold & wish You the best to finish this up untill xmas! To star new year with all the power under You fingertips in Octane

[Tutor]

Glimpse, Thank you for all of your assistance.

Hi, Tutor!

Your radiator pick is awesome! MoRa 9x120mm + aquacomputer 240mm is nice step.

I picked the MoRa's and will connect them in line before Mr. Freeze(r) to help get temps towards room ambient before coolant enters Mr. Freeze(r). Luckily, Mr. Freeze(r) can be set to 0 degrees Fahrenheit or -17.77 Celsius. Mainly when the Tyan system isn't on, i.e., I'm not intending to start it up soon, I'll just set Mr. Freeze(r) to maintain a refrigerated temperature of between 40 to 45 degrees Fahrenheit.

So, You going to end up cooling 4x TitanZs or more? =)

I have 6x Titan Z Hydros and 1x Titan Black Hydro to cool down.

Even for 4x TitanZs these two MoRas should be enough to keep the temperature 10-15 C higher then ambient.

I'm counting on their being able to do, at least, the same with 6x Titan Z Hydros and 1x Titan Black Hydro.

Now I migt be wrong, but those rads in fridges are not going to help, but make that fridge act like an oven I want to be wrong , but I migh be right I don't believe thei are going to cope with heat from rad. Especially keeping in mind how much air is in them

I too have considered that there will be a battle of heat vs. cold. Thankfully, Mr. Freeze(r) is a mini and it cost only about $100 (US). It's rated at having an annual energy cost of only $31 (US). A mod that I'm also planning for Mr. Freeze(r), if heat appears to be the victor, is to release heat by installing a few small diameter holes in the top of the Mr. Freeze(r) and running a narrow diameter plastic pipe through each of those holes into the refrigerated compartment and placing small, very light vent caps on top of the pipes so that heat can escape through them. But before doing this however, I plan to develop a pre-start up regime for the system where I can better pinpoint how long before I boot the Tyan must I pre-chill the coolant and lower the temperature of Mr. Freeze(r), and thus the coolant, so that when I first boot the Tyan the effect of Mr. Freeze(r) dominates, as a whole, throughout my computing experience. Also thankfully, as a last resort, of course, Mr. Freeze(r) does have a door that I can easily open if heat appears to be the victor and even with the door open to various degrees, Mr. Freeze(r) might still be able to lower temperature to a noticeable degree. Also, to aid my testing and use, I'm installing temperature sensors before coolant enters into the MoRas, immediately after the coolant reaches the exit of the last MoRa and a third sensor after the coolant leaves Mr. Freeze(r).

What is the ambient temperature of Your room going to be? Hope it's pretty low - the best case scenario i believe You can get is no lower temps than ambient (if those freezers are not going to fail)..but hey it's nice experiment & I hope I'm wrong

The ambient temperature of the room where this system will be located is usually between 65 to 75 degrees Fahrenheit. I'm not going to bet my life that this experiment will be a complete success. At age 61, I like a good challenge. If Mr. Freeze(r) fails it won't fundamentally affect me financially. But if I can get this to work a significant extent, such as by getting the temps to between 10 to 20 degrees below ambient, I wouldn't feel it was a total failure. Moreover, I 'll have quick disconnects throughout the system to help modify it as needed. But even more importantly, even if I declare the experiment a total failure and I have to move all of the cooling units from Mr. Freeze(r) to being just room ambient air-cooled, I definitely will not feel like a failure because I gave it my best try by using all that I've learned in my shell's short span on this side of the ground.

Really curious how it's going to unfold & wish You the best to finish this up untill xmas! To star new year with all the power under You fingertips in Octane

I will keep all of you posted on how this project develops. If you think of or hear or read about anything that might help make this experiment of mine a success, please let me know.

Thanks always for your best wishes and for your kind aid. I love all of my cousins no matter how far and for how long we've been scattered across the face of the earth - know that that includes you. May every day bring success to you and all of your loved ones beyond your wildest dreams and imaginations.

[Tutor]

Update: It won't be finished by Xmas because some important parts will not be arriving any earlier than the day before Xmas, and may not arrive until after the 25th. Thus, my new projected deadline is December 31st.

[smicha]

Tutor,

First of all - hats off, respect!

What fans do you have for the double Aquacomputer radiators? What are their static pressure? Pushing air through such thick fans require about 3mm H2O static pressure fans (or even more). Just asking.

More and more great machines on the forum