Few questions about smicha's watercooled 7 gpu build Tweet

[jeunack]

Hi everyone,

I'm French, so sorry for my english mistakes

I'm motion designer and associate in a Digital Agency, based in Lyon, France.

I love Octane, I work with a dual 980ti in my workstation with Cinema 4D, I want to invest on a Water Cooled Render Farm with 7x1080ti GPU, for rendering in octane (of course) and use both for "live preview" through Ethernet Network.

I'm really impress on what smicha do on his last build-log with 7 GPUs 1080gtx

I invite you to check on his youtube channel for details https://www.youtube.com/user/smicha7/videos

and the configuration :

http://www.overclock.net/t/1618497/build-log-14x-gtx-1080-watercooled-octane-render-network-workstation

really impressive !!

but after read many topics, I still have few questions.
I could directly ask to smicha, but I prefer post here, maybe can be useful for someone

So my questions are :

- Is it really necessary to get 2 xeon cpu to drive 7 gpu, only for use octane render as slave, or an i5 / i7 CPU is enough ?
In this case its very difficult to find a recent motherboard with 7 slots PCIe and a Single CPU...

- I think 1080 and 1080ti are very closed, but Smicha's build log still compatible with 1080ti GPU ?

- I don't have the same skill to cut and i/o bracket like smicha but if I buy a gpu without DVI port like the MSI GeForce GTX 1080 Ti AERO 11G OC and replace with this single slot bracket : https://shop.aquacomputer.de/product_info.php?language=en&products_id=3657 would be ok ?

- 1 gbps LAN is it enough or I need a 10Gbits LAN between my workstation and the cluster, to avoid slowing render through the network ?

- I read that it's pretty difficult for the motherboard to recognize 7 GPU, and smicha says that a custom Bios is required ?
Is it difficult to set up ?

Thanks for your answers

[glimpse]

Hi, Jeunack

- Is it really necessary to get 2 xeon cpu to drive 7 gpu, only for use octane render as slave, or an i5 / i7 CPU is enough ?
In this case its very difficult to find a recent motherboard with 7 slots PCIe and a Single CPU...

no it isn't. 7 GPU build could be build using single CPU motherboards, but there are only few of them on the market. First 7GPU build that Sebastian made actually was single CPU based - take a look.

- I think 1080 and 1080ti are very closed, but Smicha's build log still compatible with 1080ti GPU ?

1080TI are significantly faster than 1080s & yeah, if You wish 7 GPU build could be configured using new cards, the only difference would be required amount of power, since 1080Tis would draw a bit more.

- I don't have the same skill to cut and i/o bracket like smicha :) but if I buy a gpu without DVI port like the MSI GeForce GTX 1080 Ti AERO 11G OC and replace with this single slot bracket : https://shop.aquacomputer.de/product_info.php?language=en&products_id=3657 would be ok ?

You're right, 1080Ti comes in single slot form factor when watercooled if You do not get specific model that has extra DVI. So no cutting is needed & simple change of I/O bracket would do the job.

- 1 gbps LAN is it enough or I need a 10Gbits LAN between my workstation and the cluster, to avoid slowing render through the network ?

1gbps lan will be the bottleneck in most cases, so getting the fastest connection would certainly help. If You're investing into motherboard that would be capable handling 7GPU, 10 gig lan will probably be the by default.

- I read that it's pretty difficult for the motherboard to recognize 7 GPU, and smicha says that a custom Bios is required ?
Is it difficult to set up ?

in bios You should enable 4g functionality & in some motherboards You better have custom bios that has that functionality enabled by default. flashing bios is pretty simple process, but getting those might take time, as not all companies are going to help anyone.

I guess Sebastian is going to give You more on those themes.

[itou31]

Bonjour jeunack,

That's cool to see a french guy with futur seven GPU rig.
And also, happy to see these octane/hardware masters answer to our foolish project (Tom, smicha, tutor and others...).

As my futur project is on the way (after checking with smicha about the motherboard) I go for the asus X99-E WS/USB3.1 (not adviced by smicha as it has many issues).
I just verifed that I can boot with seven GPU (just test with 1080ti + 1080 + 980ti + 980ti + 580 + 2 AMD hd 4550). Not go further into windows. But at least it boots fine and see all boards in BIOS.

So my rig will have 2x1080ti + 3x980ti + 2xTitanBlack.

For network performance, as I tested (1Gigabit LAN), its performs better than GPU connected using riser (USB3 on PCIe1x) or cluster/splitter (4 GPU to 4xPCIe). All this performance issue is due to the V3 kernel with many communication with host RAM (frame buffer and tonnemapping). I can only see a performance decrease about 3 to 5% when network rendering, as opposed to 10% to 50% when using risers (with 1080ti).

And I think that with 10G it will still be better, but a bit expensive too compare to the performance gain (just an assumption).
So let we know your project...

[jeunack]

Bonjour tout le monde :p

Thanks for your answers glimpse and itou31 !

That's cool to see a french guy with futur seven GPU rig

Yes, I hope I could build a GPU rig like this thanks to you !

glimse, your article is really helpful, thanks ! Really !

Now I'm sure that 7 gpu directly plugged on the motherboard, without risers or split, are the best option for me.

1gbps lan will be the bottleneck in most cases, so getting the fastest connection would certainly help. If You're investing into motherboard that would be capable handling 7GPU, 10 G lan will probably be the by default.

It's not very simple to find a 7pcie motherboard with 10G LAN, and I'm not sure that the performance are really significiant, and to be honest, when you see the price of a small bussiness 10Gbits LAN equipment for 8 people in my agency, you prefer invest on a second gpu rig and stay with 1gbits LAN

As my futur project is on the way (after checking with smicha about the motherboard) I go for the asus X99-E WS/USB3.1 (not adviced by smicha as it has many issues).
I just verifed that I can boot with seven GPU (just test with 1080ti + 1080 + 980ti + 980ti + 580 + 2 AMD hd 4550). Not go further into windows. But at least it boots fine and see all boards in BIOS.

I had the same idea to choose the motherboard, I just want to be sure that asus X99-E WS could be run 7 GPU on windows 10, before deciding to purchase this build

So it would be my future build GPU rig based on smicha's advise with his watercooling :

MSI GeForce GTX 1080 Ti AERO 11G OC x7
Carte mère Asus X99-E WS
Intel Core i7-5820K (3.3 GHz)
Thermaltake Core X9
EVGA SuperNOVA 1600W TITANIUM
Samsung SSD 850 EVO 500 Go
Corsair Vengeance LED Series 32 Go (4x 8 Go) DDR4 3200 MHz CL16

Let me knows if you see something wrong !

[FrankPooleFloating]

The i7 5820k only has 24 lanes. If you get a 5960x you'll get 40 lanes, and presumably with 7 GPUs your first one would be 8x and the others would be 4x. I am not up on the latest Intel CPUs, since I have not been in the market for one lately, but maybe some other new ones have 40 PCI-E lanes...

[jeunack]

The i7 5820k only has 24 lanes. If you get a 5960x you'll get 40 lanes, and presumably with 7 GPUs your first one would be 8x and the others would be 4x. I am not up on the latest Intel CPUs, since I have not been in the market for one lately, but maybe some other new ones have 40 PCI-E lanes...

You're right ! thanks ! The i7 5930K should be good

I will send an email to Smicha, to knows if it's ok with this Mobo and the Bios to drive 7 gpu

I'll let you know

[smicha]

Thank you all for all help to the build.

Some notes:

1. Even a 28 laned cpu is fine with dual PLX mobo for 7 gpus
2. The only drawback of i7 mobo as asus is lack of asus support for custom bios with 4G enabled that is why I go with dual xeon and Asrock/Supermicro.
3. Dual xeon give 80 lanes so there is no risk of PLX failure.
4. Aquacomputer single brackets are fine - I am waiting also for their delivery

Going back to unpack fresh delivery with tons of watercooling stuff - stay tuned

[FrankPooleFloating]

Right. 28 lanes... duh.

[glimpse]

pc_full.62.png

@ITOU31, one thing..keep in mind REUSED AIR. What do I mean about that, well, with 360 in front that 240 on top would loose quite a bit of it's efficiency. I made some tests with 240 front + 240 top (that reuse air from front rad) & in case of closed case there is quite a bit of a difference.

Honestly, I would really consider MORA rad - one external piece instead of the mess with multiple small rads, but I see why some might not be so much into it. Just an idea ;) ignore if it seems stupid for any reason.

[itou31]

The attachment pc_full.62.png is no longer available

@ITOU31, one thing..keep in mind REUSED AIR. What do I mean about that, well, with 360 in front that 240 on top would loose quite a bit of it's efficiency. I made some tests with 240 front + 240 top (that reuse air from front rad) & in case of closed case there is quite a bit of a difference.

Honestly, I would really consider MORA rad - one external piece instead of the mess with multiple small rads, but I see why some might not be so much into it. Just an idea ignore if it seems stupid for any reason.

Yes, you're right, I should consider low pressure inside the case for better "cooling", so I think to manage more airflow on extract (2 fans on top and one rear).
Else I have a big phobya 1080 on the other side with some low airflow fans.