Learning about water cooling 780 Ti cards

[EngineHouseVFX]

Starting a new thread so not to just hijack rappet's thread (http://render.otoy.com/forum/viewtopic.php?f=9&t=43119)

We are regularly using 3 machines with a total of 11x 780 Ti SC cards with Octane. We were regularly getting overheating problems so used the EVGA precision software to throttle them to reduce temps, which kinda defeats buying the super clocked 780 cards.

We tried adding fans and whatnot but some of the cards are getting really hot, the middle cards sitting at about 96 degrees C during render time and getting regular render drop outs and errors.

So it's water cooling time! I built the machines and it was my first build, I know nothing about water cooling.

Here's the first machine:-
3x EVGA GeForce GTX 780 Ti Superclocked ACX Cooler 3GB GDDR5 (03G-P4-2884)
1500W PSU

In a phantom 530 case


Plus here's the second machine with the netstor:-



We'll aim to solve those first, the

Replies so far:-

GIGABYTE GA-X79-UP4 (rev. 1.0) motherboard
3x EVGA GeForce GTX 780 Ti Superclocked ACX Cooler 3GB GDDR5 (03G-P4-2884)
1500W PSU

In a phantom 530 case

it seems Your cards have ref designed pcb : http://www.coolingconfigurator.com/step ... _gpus=1262
so, there's plenty of water blocks to choose from =) unlike when You have something custom, like STRIX from ASUS =)

as for case You have enough space to play with =) 240mm in front & 360 up top (look for some case reviews what modders recomend & if case doesn't have any compatibility issues =) Then get pair of rads, 5x Slow spinning, pressure optimised fans (like noctua nfp12); pump & res (or combo =) with something like d5; some tube + fittings & You're ready to go =)

Make new thread if You need more assistance & I believe some guys will be happy help You out =) ACX (from EVGA), WINDFORCE (from Gigabyte), etc. are not the best coolers in for multiGPU rigs, plus they throw a lot of heat back into the case, rising up ambient temps & dropping efficiency of cooling that will effect performance & more then likely throttle down GPU's (loss of performance comes along..)

the loop will cost You a bit, but You probably get performance of extra card too, less temperature & less noise too =)

EK-FC780 GTX Ti - Acetal+Nickel [3831109868645] covers all pcb which is great for dust removal, unlike short waterblocks. Get a backplate - great investment - not for looks only but for temps.

I am afraid your case has not enough radiator capacity. For 3xgpu and cpu I'd go with 2x480 or some 3x3 external solutions - Glimpse please help with the link to it.

Agree with Smicha =) all he way, & here's the link for free standing 9x 120 or 9 x 140 MoRa rads:
http://watercool.de/en/mo-ra3

here are some photos, I've shared on the other thread =) http://render.otoy.com/forum/viewtopic. ... 1&p=208691

p.s. 240+360 might be enough to cool down three cards, but temps will not be ~40C, more or less 60+C (depending on how fast Your cards will be clocked) Free standing rad might be better solution (at least I'd choose that route, simply 'cos You have more expandability & could reuse later =)

I'm not worried about CPU cooling, do you have a guide page to what 480 / 140 / 120 & MoRa rads are?

[glimpse]

I'm not worried about CPU cooling, do you have a guide page to what 480 / 140 / 120 & MoRa rads are?

I'll start from explaining some terminology before making a proposal of how to deal with entire loop (full list of parts, routing & expected performance)

Usually fans are in two sizes 120mm & 140mm, Though You can find (50mm, 80mm, 92?, 180, 200,230 etc.) however most case manufacturers chose to use 120/140 - any big aftermarket water-cooling solutions will bring most value at these sizes.

240,360, 480 is simply multiplied 120mm size (2x,3x,4x..etc.) the same goes 280, 420 (as multiplied from 140mm) etc - You get the idea..

MoRa (check the link) is a monstrous free standing radiator - it has two models 9x 120mm (4x 180mm) or 9x 140mm. for those who don't have enough space in the case You can hook external radiator without need of placing them inside Your case (some are just too small =)

now the size doesn't matter most (it's not a secret =) there are other things like fin density (that influence efficiency) & required optimal pressure (to choose fans), build quality, resistance (to choose pumps) & other things..- there is ton of info, but it's a waste of time to read trough everything for one or two builds =)..

We'll try to give You some ideas to choose from =) I just need to do some homework about what You have =)..

P.S. what temps You're getting in Netstor box? it has some gaps to breath, though Your cards dump air back to the case (non reference design coolers =) check this thread for some inspiration =DDD

[smicha]

That's right - tell us what you have. Do you want to watercool netstor gpus? What are these gpus?

[EngineHouseVFX]

Parts are as described above, 2 machines are the main concern at the moment

Machine 1:-

GIGABYTE GA-X79-UP4 (rev. 1.0) motherboard
3x EVGA GeForce GTX 780 Ti Superclocked ACX Cooler 3GB GDDR5 (03G-P4-2884)
1500W PSU
In a phantom 530 case


Machine 2:-
GIGABYTE GA-X79-UP4 (rev. 1.0) motherboard
6x EVGA GeForce GTX 780 Ti Superclocked ACX Cooler 3GB GDDR5 (03G-P4-2884)
1500W PSU
2xGPU In a phantom 530 case and 4xGPU in the 1200w Netstor box

As I'm sure you'd expect it's the middle cards that are the hottest, they were hitting 95 C and above before we started throttling them, now we're running them at 70% and the temps are about 85. The Netstor cards run normally with the jumper settings for full fan speed and they sit at around 60-70 C.

[smicha]

Short question - can you sell all gpus, one motherboard and buy 4 titans Z ? You'll have power of 11 titans when you watercool them.

[glimpse]

Short question - can you sell all gpus, one motherboard and buy 4 titans Z ? You'll have power of 11 titans when you watercool them.

don't think they're going to do that =) but I see You point - cooling all of these will cost twice as much as having to cool only 4x Zs =)


so let's start, with the rig that holds 3 cards, EngineHouseVFX. I'll drop a line in half an hour =) putting down some ideas into text =)

[rappet]

Hi guys,
You need certain types of mobo for watercooling, becuase of space for the backplates ant what other reasons?
I guess with this mobo it impossible to stuck 3xTitanZ on watercooling... You need the backplate, and that is not possible in the upper slot close the the ramm. (Right now I have a Titan Black without backplate in the upper slot)
Greetz,

[smicha]

You need the backplate, and that is not possible in the upper slot close the the ramm. (Right now I have a Titan Black without backplate in the upper slot)
Greetz,

But EK back-plate and their water-blocks fits perfectly well. This is only an issue for stock back-plate.

[glimpse]

I guess with this mobo it impossible to stuck 3xTitanZ on watercooling... You need the backplate, and that is not possible in the upper
Greetz,

EK gives out bracket for TitanZ to convert card from 3slot to dual slot =) & that means You can house up to 4x cards on the regular motherboard with dual slot spacing between PCIe connectors =) so technically 8 GPUs (4x dual GPU cards) are possible in relatively compact case =)

[glimpse]

so, let’s start with Machine 1, @EngineHouseVFX :

(GA-X79-UP4; 3x EVGA GeForce GTX 780 Ti Superclocked ACX Cooler 3GB, 1500W PSU in a Phantom 530 case)

From photo I see You have HDD/SSD cage - I'd remove it completely & get the SSD on the back of the motherboard tray (there’s a spot on the other side) or place it in non-removable optical drive cage.

In front I'd place thick 240mm (2x120mm) RAD with good fans (You'll find recommendations bellow), nearby You’d have space for PUMP & RES. I would recommend combo in one unit (bought this model for myself lately =), to avoid unnesecary fittings & tubing..

(there are other option that almost every manufacture has & You can choose bay reservoirs /with pump or without/ - but those are hard to refill + then You need pump alone if it's not included & with pump inside vibration might go into case, making some audible buzz.. Keep in mind, more components You’ll have, more cluttered Your case will be & all these joins will bring extra place for user error..)

So You get 240 RAD with in with fans, then PUMP/RES combo, from here TUBE should go to GPUs (water blocks). After those I’d put 360 RAD (3x120mm) on ceilings. Mounting fans on top of the frame will give You a possibility to mount quite thick radiator bellow. From this top mounted RAD TUBE would go to first 240 RAD & the LOOP would start again. Simple! =)

Having 360+240 is quite enough for 3x 780Ti - even with OC (& extra performance that comes along) they should run ~60C. It all depends on Your radiators. If You get thick enough RADs & good pressure optimised FANS, noise should be minimised, but You’ll never get throttling down because of overheating..& Your system will be more stable =)

To keep things out of dust I would spin three top fans at low RPM while those on the 240 a bit faster. also would add extra fan in the bottom to keep positive pressure & intakes dust filtered (depending on the way You’re going to mount pump/res that might be possible or not). Maybe I would flip the fan on the back to draw air in (feading CPU cooler with fresh air) - don't forget to dust filter if You choose so.

Overall the case is quite friendly, but there are some glitches here & there, that You need to think of if building a loop inside..- like airflow obstacles & such.. =)

For the fans, I’d personally use Noctua: there are there lines: “Original” (brown/beige), Redux (grey), & server..- Redux is least expensive, though Server fans will give You wider range of RPM’s (up to 3000RPMs if You ever need). Though if You choose right rads Original should fit perfectly - they run very silent & performance is the best (haven't seen a single negative review on those - & it's well worth to pay 30$/fan)

For the rig #2 id do almost the same, but bit differently, & I'll cover that a bit later today =)

If You have any questions: drop a line back & I'll try to explain things out =)