Feedback on proposed 13 GPU rendering machine build

[ValentinMoebs]

Hello all,

I am posting this message because I am looking to get some feedback from people more experienced than I with multi-GPU builds.

I am currently working on a machine with an RTX 2070 and a GTX 1080 Ti, and am rendering quite a bit of video. This regularly results in spending directed towards render farms in order to meet deadlines. In the long run, it will be undeniably more cost-effective to build a machine locally dedicated to GPU rendering in order to avoid having to farm out frames. Following is my proposed build. If you have any experience with muti-gpu systems and mining hardware, I'd be very interested in your opinion on this. The two main goals for this build are cost-effectiveness (which led to the choice of RTX 2060 SUPER GPUs) and ease of use for launching renders (single node).

I intend to run Windows 10, C4D and Octane on this.


Here is the prospective component choice :

CASE :
1x Nanoxia Hydra VI
(Mining case made to accomodate 13 dual-slot GPUs, sold without risers)
https://www.nanoxiausa.com/products/hyd ... 3-gpu-case

MOTHERBOARD :
1x Asrock H110 Pro BTC+
(Mining motherboard with 1 x16 PCI-E slots and 12 x1 PCI-E slots - socket LGA 1151)
https://www.amazon.fr/Asrock-H110-Intel ... op?ie=UTF8#

RAM :
1x Crucial CT2K16G4DFD824A 32Go Kit
(Kit of two 16 GB sticks, 2400MTs non-ECC)
https://www.amazon.fr/Crucial-CT2K8G4DF ... 64&sr=8-20#

CPU :
1x Intel Core i7-6700K
(In the top end of what is supported by LGA 1151 and that is available reasonably quickly)
https://www.amazon.fr/Intel-Skylake-Pro ... 701&sr=8-1

GPU RISERS :
13x LONGXI 6-pin PCI-E 1x to 16x riser
(Seems to be the exact same product, albeit rebranded, as the one distributed by Nanoxia for use in their cases. Works using a USB3 cable, which is convenient in the case that it turns out that the cable is too short)
https://www.amazon.fr/LONGXI-6-pin-Expr ... NrPXRydWU=

GPU :
13x ZOTAC Gaming GeForce RTX 2060 Super Mini Edition
(Seems to be the most cost-effective card on the market right now with decent OctaneBench scores)
https://www.amazon.fr/Carte-Graphique-Z ... 41&sr=8-23#

PSU :
1x IBEST IMPETUS d'alimentation ATX 1600 W Mining PSU
(For supplying power to the MB and the first 6 of the GPUs)
https://www.amazon.fr/IBEST-IMPETUS-dal ... 97&sr=8-16

1x lzndeal Antminer 1600W/1800W APW3 PSU
(For supplying power to the rest of the GPUs)
https://www.amazon.fr/lzndeal-Antminer- ... 46&sr=8-19


As far as storage goes, I have a few drives lying around, so I plan to use one of those, but this machine will mostly be retrieving from and saving to a NAS.

If any of you can foresee compatibility problems with this build and potentially recommend alternative hardware, then I would very much appreciate it.

Many thanks in advance !

[Antoncromas]

You've picked a discontinued CPU with a total of 16 PCIE lanes, 1 video card takes 16 lanes for the fastest data transfer, you do the math. Your machine will eat some of those lanes for hard drives now let's say you have 12 lanes left and 13 cards? That build will be lucky to be able to run 3 cards at 4x2x2.

Option 1 - You will need to start here https://www.supermicro.com/products/sys ... GR-TRT.cfm but you will be looking at probably 15k+ investments at which point renderfarms start to look pretty good again =D

OPtion 2 - Get the new AMD threadripper that has 88 PCIE lanes, you should be able to run 8 cards just fine, use PCIE extenders https://www.newegg.com/p/N82E1681218305 ... -_-Product . This system will be a bit cheaper but you will likely end up dropping 8k+ on it.

Anton

[ValentinMoebs]

Hey Anton,

Thanks for your reply. The motherboard I selected has 12 1x and 1 16x PCIE ports, so I did expect to barely have enough lanes (using only 1x lane in the 16x port) despite using this MB + CPU. Despite this, the second option you are putting forward is interesting even if it does entail spending a bit more - I will investigate and get back to this thread.

[Lewis]

Also bare in mind that GPUs on 1x slot (via extenders) will be slower (in my tests 25-35% depending on extenders and number of GPUs) and you will loose quite bit of render performance.
Since octane v4 that speed drop is much more noticeable than with octane 2-3.x since now frame buffer is at CPU/system ram so GPUs communicate with system much more often.

[Antoncromas]

You really don't want to use a mining rig setup for a render station, you'll be pulling your hair out trying to troubleshoot it when something doesn't work.

The reason you will need PCIE extenders is because each card will take 2 slots so you would need to have an open air type rig with cards sitting above the motherboard. There is also a way to make them take a single slot but that involves custom waterloop and it becomes a real pain to swap the cards.

You can get 8 better cards for the same price as 13 you originally wanted and the rig would run way better when the system is not choked up with a lack of PCIE lanes.

If you are in a rush you can always go with a current gen threadripper, that will only offer 64 lanes but should be a little cheaper as well.

[glimpse]

Also bare in mind that GPUs on 1x slot (via extenders) will be slower (in my tests 25-35% depending on extenders and number of GPUs) and you will loose quite bit of render performance.
Since octane v4 that speed drop is much more noticeable than with octane 2-3.x since now frame buffer is at CPU/system ram so GPUs communicate with system much more often.

actually for OB the impact is not that big when using Gen3 x1. Think second fastest machine in OctaneBench, the one with 17x 1080tis is actually build using splitters and x1 risers. If done right, You can get stability and speed, if You cut corners then it will not be fun working on that..

[Lewis]

actually for OB the impact is not that big when using Gen3 x1. Think second fastest machine in OctaneBench, the one with 17x 1080tis is actually build using splitters and x1 risers. If done right, You can get stability and speed, if You cut corners then it will not be fun working on that..

Hi Glimpse (Tom),

Well i tested a lot of 1x risers (Gen 2 and Gen3, Gen 4 and even those more pricey Asrock BTC Pro ones with dual sata transfer cables) and none of them i'd call "fast". Also they all experience speed loss very noticable (soem more osme less but visible, especially at big amount of GPUs, it accumulates quickly). Here is my tests from 2017:
viewtopic.php?f=23&t=56062

Needles to say that i ended up using Thermaltake 16x to 16x shielded risers for perfect speed and stability but they are very expensive (for a cable).

[ValentinMoebs]

Hello all,

Thanks for all of your replies. I investigated a little, and I could not seem to find any Threadripper boards on the market with more than 6 PCI-E slots, which would mean using a bunch of PCI-E splitters if I wanted to reach the originally desired number of cards.

As far as using less but more powerful GPUs, that won't be as cost-effective I'm afraid. I'd rather build a second system later with more cheap cards than splurge on more expensive cards now. Using available price and Octanebench information, I've found that RTX 2060 S cards can offer a ratio of about 2 € per Octanebench point whereas the next best RTX 2070 S only offer 2.45 € per Octanebench point.

After looking at Threadripper motherboards for a while, I remembered that I have an Asus Z9PE-D8 WS board currently underutilized in another machine that I could use for this project. Both CPU sockets are populated, which means iirc that a total of 80 PCI-E lanes are available. I am including the block diagram for the motherboard below :



As it is, the board has 7 x16 physical slots, but some of them, like slots 1 thru 4 are already connected to switches. Slots 5 thru 7 however, are not connected to switches. Using an Amfeltec x4 to 4 x x16 PCI-E splitter on that last slot and risers on all others could allow me to potentially connect 10 GPUs. I have read threads on this forum where people talked about successfully using these splitters, and so while they are pricey, they seem like a safer bet than another brand.

https://www.amazon.fr/voies-X4-PCIe-ver ... 156&sr=8-3

Does this seem like a reasonable proposition ?

[Antoncromas]

Hey,

Yes most boards now have only 4 x16 PCIE slots sadly. That may change when the latest 88 lane chip from AMD comes out but I doubt it.

Most elegant solution is to use x4 external boxes likes of Cubix. You can get them pretty cheap on ebay and they have capacity of storing 4-8 GPUs and plug into your motherboard PCIe ( I believe its a PCIe card from cubix with external connectors)

https://www.ebay.com/itm/Cubix-XPDT-X16 ... Sw7PNd3a0I

https://www.ebay.com/itm/Cubix-GPU-Xpan ... SwGEJcZIh5

https://www.cubix.com/xpander/

My coworker has been using the x4 box successfully, I have not tried it myself however.

Hope that helps.

Regards,
Anton