Best Practices For Building A Multiple GPU System

[smicha]

Hello Smicha,

I especially like your Youtube video Part 24.

Which motherboard and case do you intend to use for the upcoming 20 GPU system?

Love it too This will be my custom case made of pure alloy. I am planning to use the Trenton 18PCIe backplane
https://www.trentonsystems.com/backplanes/bpg8032 - hope I get it soon. I have no idea if I manage to run them but at least I must give it a shot

[Tutor]

Maybe priced has dropped from about -
(Per Boris [ - viewtopic.php?f=9&t=26097&start=10#p121261 ] circa 2013)
"expansion kit (host card, target card, 3m cable): $1426
bgp8032 backplane: $3078
shipping to europe fedex international priority (deadline!!): you do not wanna know."
= $4,504 for backplane/expansion board and interface kit. However, that would be a big surprise to me since the market demand is possibly somewhat greater now if it works as we desire it to work with Octane. Assuming it does work, contrast that price total with the price of Amfeltec's 4 x SKU-042-43 GPU oriented PCIe 4-way Splitter = $ 223.49 US each or $893.96 total for four of them. Thus, the $3,610.04 excess is for x16 vs x1 PCIe card connections (using Amfeltec splitters) and a ready-made location for 17 single spaced water cooled GPUs like those in your videos. Choices, choices!

[smicha]

Maybe priced has dropped from about -
(Per Boris [ - viewtopic.php?f=9&t=26097&start=10#p121261 ] circa 2013)
"expansion kit (host card, target card, 3m cable): $1426
bgp8032 backplane: $3078
shipping to europe fedex international priority (deadline!!): you do not wanna know."
= $4,504 for backplane/expansion board and interface kit. However, that would be a big surprise to me since the market demand is possibly somewhat greater now if it works as we desire it to work with Octane.

I have one guy to go with it. So soon I will give it a shot. Thank you so much, as always!

[Tutor]

When you say, "I have one guy to go with it.", is his system using 17 single spaced water-cooled GPUs like those in your videos on a Trenton 8032 backplane and the other 3 GPUs on a traditional motherboard (which one?) (for a total of 20 GPUs)? What level of performance is that system achieving? What OS is it using, Windows or Linux (and which version and level no.)?

[smicha]

When you say, "I have one guy to go with it.", did his system use 17 single spaced water-cooled GPUs like those in your videos on a Trenton 8032 backplane and the other 3 GPUs on the traditional motherboard (for a total of 20 GPUs)? What level of performance is that system achieving?

I meant - I have one guy interested in having the 20 gpu-based machine. And the idea is to use 16 cut gpus on the backplane, 1 PCIe slot has to be reserved for the link, 4 cut gpus on a host dual xeon E5 motherboard with one PCIe slot free for the link. 3x1600W PSUs. Custom case. Please share your thoughts. For now my concern is OS that could handle 20 GPUs...

PS and yes - all will be watercooled.

[Tutor]

A given which we must account for is the software the user intends to deploy. Only certain 3d applications now run under linux whereas all of them run under Windows. The big "however" is that - Can Windows support 20 GPUs and all that I've seen from my somewhat limited experiences is that the answer is "No, only Linux can do that." However, Linux systems can be used in renderfarms when one uses a Windows or Mac controller system and it/Linux can render from Windows or Mac based applications using render farm management software like Deadline [ http://www.thinkboxsoftware.com ] - thats where I'm headed as I intend to have, at least, five Octane massive (i.e. ~ 18-20 GPU Linux systems) in my personal render farm working under the command a four GPU MacPro 2012+ and a twelve GPU Windows system. As I now have only seven Octane licenses, I 'll soon be adding more of them and related Octane plugins to my wolfpack.

[smicha]

A given which we must account for is the software the user intends to deploy. Only certain 3d applications now run under linux whereas all of them run under Windows. The big "however" is that - Can Windows support 20 GPUs and all that I seen from my somewhat limited experiences is that the answer is "No, only Linux can do that."

If Linux can do that, I love Linux immediately

[Tutor]

Please see my last post in about 10 minutes from now, as I'm editing it.

[smicha]

Please see my last post in about 10 minutes from now, as I'm editing it.

Thank you . I am watching Deadline videos now. So is the Deadline a solution to handle many workstations under one windows based host for Octane?

[Tutor]

Licensing and Application/OSes linkages Matter When Discussing Many Workstation Environments

Please see my last post in about 10 minutes from now, as I'm editing it.

Thank you . I am watching Deadline videos now. So is the Deadline a solution to handle many workstations under one windows based host for Octane?

Sort/Kind of - because application licensing comes into play in a render farm environment. The understanding that I have of Octane's licensing structure is that there has to be an Octane main license for each machine (and the applicable plug-in license for each machine if one is rendering in the plugin environment). Also, here's some information that I put together for some major GPU renderers showing supported 3d applications and OSes. It's obvious that Octane leads the way in breathe of multi-OS coverage.

OSes and Major 3d GPU 3d Apps currently supported by some 3d GPU Renderers [as of Jan. 2017] are:

1) RedShift:
Maya - Windows & Linux
Houdini - Windows & Linux
Softimage - Windows - EOL
3d Max - Windows

2) TheaRender
Blender - Windows & Linux & MacOS
Cinema4d - Windows & MacOS
FormZ - Windows & MacOS
SketchUp - Windows & MacOS
Fusion 360 - Windows & MacOS
3d Max - Windows
Rhino - Windows

3) FurryBall
Maya - Windows
Cinema4d - Windows
3d Max - Windows and

4) Octane:
Maya - Windows & Linux & MacOS
Blender - Windows & Linux & MacOS
Modo - Windows & Linux & MacOS
Houdini - Windows & Linux & MacOS
Cinema4d - Windows & MacOS
Poser - Windows & MacOS
Nuke - Windows
3d Max - Windows
Softimage - Windows - EOL

Because at this stage I'm overly cautious, I've opted to favor the following applications as potential best choices for running an individual's massively populated Linux systems for 3d rendering because they support all three of the most popular OSes (and in the case you confront - Windows) and Octane has increasingly become the leader in it's breathe of support:

1) Blender - Windows & Linux & MacOS
2 Houdini - Windows & Linux & MacOS
3) Maya - Windows & Linux & MacOS
4) Modo - Windows & Linux & MacOS

So, one of these four applications would be my preference for running on a Linux render farm managed by Deadline [or just on a single 20 GPU Linux system]. Among those four - Blender's the only free one; contrarily, there's the challenge of staying clean under the software licenses of the other three 3d products, especially in a render farm environment.

In sum, my preference for the render farm as a whole is leaning towards running a free application, i.e., Blender, with reasonably priced Octane plugins, under a free OS (Linux Mint) that can win the fight against "Out of IO space" issues/messages (if you're lucky enough to get coherent messages).

P.S. All of the above discussion is not to say that one cannot send a scene/animation rendered under one nonLinux based application, such as C4d, to a Linux server and have it successfully rendered there. My bet is that almost, if not, all cloud rendering facilities are Linux based. I just like the comfort of having the same applications being able to be run at both ends - on the server & on the slave.