Best Practices For Building A Multiple GPU System

Discuss anything you like on this forum.
Post Reply
User avatar
Notiusweb
Licensed Customer
Posts: 1285
Joined: Mon Nov 10, 2014 4:51 am

Smicha!
Nice job and congratulations!!!
Wowwy-Kuzzowwy!!!

Tutor helped me through the woods many times, he is the most valuable resource I have ever come across with regards to rigs, and health drinks 8-)

Fascinating as well. I myself have experimented lately with PBR shaders more and more, and when used in Octane (via a script), they seem to require less compute time. Meaning it takes less samples to reach a rendered look (gloss without artifacts, graininess, or hot pixels...) . Along with the potential of having a closer to real-time solution in Brigade one day (per discussions of its ability), renders will have the potential to go much faster by default.
Yet, monster-rigs I think will still be valuable as UHD and VR takes over HD, and our rendering requirements escalate as well.

But nice work, much credit, and happy to see you proud of the accomplishment!
Definitely this is to celebrate, Smicha. And by 'celebrate' I mean manipulate the heck out of everything to boost the score even higher!....

Never forget, Tutor's Volta rig is on the Horizon...

The Render-Coats are coming....
The Render-Coats are coming....
The Render-Coats are coming....
Win 10 Pro 64, Xeon E5-2687W v2 (8x 3.40GHz), G.Skill 64 GB DDR3-2400, ASRock X79 Extreme 11
Mobo: 1 Titan RTX, 1 Titan Xp
External: 6 Titan X Pascal, 2 GTX Titan X
Plugs: Enterprise
User avatar
Tutor
Licensed Customer
Posts: 531
Joined: Tue Nov 20, 2012 2:57 pm
Location: Suburb of Birmingham, AL - Home of the Birmingham Civil Rights Institute

Hello Notius,
Rather than buy Voltas, I think I should buy more Nvidia - stock.
Because I have 180+ GPU processers in 16 tweaked/multiOS systems - Character limit prevents detailed stats.
User avatar
smicha
Licensed Customer
Posts: 3151
Joined: Wed Sep 21, 2011 4:13 pm
Location: Warsaw, Poland

Thank you Notiusweb.
3090, Titan, Quadro, Xeon Scalable Supermicro, 768GB RAM; Sketchup Pro, Classical Architecture.
Custom alloy powder coated laser cut cases, Autodesk metal-sheet 3D modelling.
build-log http://render.otoy.com/forum/viewtopic.php?f=9&t=42540
User avatar
Notiusweb
Licensed Customer
Posts: 1285
Joined: Mon Nov 10, 2014 4:51 am

Thoughts on the Voltas?
They look great, although I was envisioning a mighty amount more when they first were teasing Pascal. I thought we would see 24GB VRAM and something like 10,000 CUDA. Not that 16GB VRAM and 5K+ CUDA is bad, just more incremental.

You know what though, this is something I have noticed. I put time into watercooling the Titan Zs because they would heat up so fast, I lost performance unless I did so. But I now have some have the 1st gen Pascal Titan Xs (not XP), running externally on 1x PCI USB 3.0 adapters, and they always are running cooler than a regular/old school Titan X attached externally, by a good 10-12 degrees. I wonder if heat won't be that much of an issue any more for an external rig with newer cards, such as Volta (I emphasize external because internal is much different....)...like, will the power efficiency increase while the heat generated decreases? :mrgreen:
Or, will they be hotter than ever?..... :twisted:
Win 10 Pro 64, Xeon E5-2687W v2 (8x 3.40GHz), G.Skill 64 GB DDR3-2400, ASRock X79 Extreme 11
Mobo: 1 Titan RTX, 1 Titan Xp
External: 6 Titan X Pascal, 2 GTX Titan X
Plugs: Enterprise
User avatar
Tutor
Licensed Customer
Posts: 531
Joined: Tue Nov 20, 2012 2:57 pm
Location: Suburb of Birmingham, AL - Home of the Birmingham Civil Rights Institute

Notiusweb wrote:Thoughts on the Voltas? ... .
They look great to me and 16GB VRAM and 5K+ CUDA, although incremental, would be fine for me. My 1070s run cool and have never even reached 60+ degrees F even when OC'ed as much as I've cared to do so. So I'd guess that you're correct that heat won't be that much of an issue any more for an external rig with newer air cooled cards, such as Voltas, unless one is trying to double the overclock on an already maximally overclocked system.
Because I have 180+ GPU processers in 16 tweaked/multiOS systems - Character limit prevents detailed stats.
thecatch
Licensed Customer
Posts: 12
Joined: Mon Jun 20, 2016 7:40 pm

Hello i'm planing to build a mult gpu workstation with the max amount GPU you can fit in to it without cutting or other modding if possible.
I have 2x1080 that are hybrid watercooled from EVGA and a 750W seasonic unit lying around. If those components can be of use that would be great, if not then i'l get everything from scratch.

The ideal answer would be the cpu + mobo for the max amount of gpu's and the needed wattage for it to work. Thanks,!
User avatar
Tutor
Licensed Customer
Posts: 531
Joined: Tue Nov 20, 2012 2:57 pm
Location: Suburb of Birmingham, AL - Home of the Birmingham Civil Rights Institute

thecatch wrote:Hello i'm planing to build a mult gpu workstation with the max amount GPU you can fit in to it without cutting or other modding if possible.
I have 2x1080 that are hybrid watercooled from EVGA and a 750W seasonic unit lying around. If those components can be of use that would be great, if not then i'l get everything from scratch.

The ideal answer would be the cpu + mobo for the max amount of gpu's and the needed wattage for it to work. Thanks,!
If you'll be using the system only for Octane rendering then you should be able to use your Seasonic PSU in almost any event. If you need more power than the Seasonic yields, then PSUs, like your Seasonic, can be used additively if other than the one powering the motherboard, they are connected by a dual multi-power supply appliance such as this - [ https://www.ebay.com/sch/i.html?_from=R ... r&_sacat=0 ], but you must install it so that it doesn't cause an electrical short. This device prevents you from always having to turn on the PSUs in the power chain serially (turning on the PSU connected only to a few of the GPUs first, then powering up the computer by next powering on the PSU that additionally powers up the motherboard) */. But please keep in mind that a few other 3d renderers stress the GPUs more than Octane does and fewer still 3d renderers also allow simultaneous GPU & CPU rendering, requiring much more power. If you are using just 2 to 4 GPUs then explore single CPU i7 motherboards with four double wide spaced PCIe slots so you'll have room to add up to 2 more GPUs in the future. I'm using a few old Gigabyte X79 systems, each in Silver Stone Raven 3 cases, each with i7s such as with i7-4820k's, for 4 GPU tasks, but none of them are using GPUs with hybrid watercooling. Using hybrid water cooling demands that you pay a lot of attention to whether the case has sufficent grill surfaces close enough to the GPUs to attach the radiators to exhaust the heat from each radiator outside of the case. If I were using hybrid GPUs I'd prefer first a case which would allow me to exhaust the GPU radiators' heat out the rear of the case at as low to ground level as possible and allowed me to induct as much cool air blowing from front to rear as possible. I don't have any particular case in mind that satisfies this first preference. However, secondly I'd then prefer a case with as much front (and top) grill space as possible to exhaust all, or as much as is possible, heat out of the front and top of the case. Here's one 8 fanner for your consideration ( https://www.newegg.com/Product/Product. ... lsrc=aw.ds ; http://www.corsair.com/en-us/obsidian-s ... r-atx-case ). But I can't vouch for it since I've never tried or seen one up close; so you should read all the user reviews and try to find one that, at least, accomodates the hybrid GPUs that you currently own.

If you want to build a system to run more than 4 GPUs in it, just let me know; but that may likely involve a lot more modding or your paying even more money for a system specifically made for that purpose, e.g., viewtopic.php?f=40&t=62487&start=10#p324880 .

*/ You connect the dual multi-power supply appliance's sata male connector to a female SATA port on your master PSU's sata cable and connect your GPU-only power supply's main male motherboard connector to the dual multi-power supply appliance's female motherboard connector. I wrap all of my dual multi-power supply appliances completely in two or three layers of electrically insulated tape and check them periodically for security.
Because I have 180+ GPU processers in 16 tweaked/multiOS systems - Character limit prevents detailed stats.
User avatar
FrankPooleFloating
Licensed Customer
Posts: 1669
Joined: Thu Nov 29, 2012 3:48 pm

Tutor wrote: I don't have any particular case in mind that satisfies this first preference. However, secondly I'd then prefer a case with as much front (and top) grill space as possible to exhaust all, or as much as is possible, heat out of the front and top of the case...
Hey Tut, have you seen Thermaltake Core X9 and Core X5?...

http://www.thermaltake.com/Chassis/Cube ... design.htm

http://www.thermaltake.com/products-mod ... C_00002806

I'm going to finally pull the trigger and get a X9 in a couple weeks. And over the holidays do some upgrading...

X9 easily takes four hybrids and cpu aio, and even multiple 480mm rads in huge loops...
Win10Pro || GA-X99-SOC-Champion || i7 5820k w/ H60 || 32GB DDR4 || 3x EVGA RTX 2070 Super Hybrid || EVGA Supernova G2 1300W || Tt Core X9 || LightWave Plug (v4 for old gigs) || Blender E-Cycles
User avatar
smicha
Licensed Customer
Posts: 3151
Joined: Wed Sep 21, 2011 4:13 pm
Location: Warsaw, Poland

Guys. My new custom case handles 4 GPUs on air with extra slot distance so no hybrids are required. It is expandable to 11 GPUs on water. I'll show more details in few days.
3090, Titan, Quadro, Xeon Scalable Supermicro, 768GB RAM; Sketchup Pro, Classical Architecture.
Custom alloy powder coated laser cut cases, Autodesk metal-sheet 3D modelling.
build-log http://render.otoy.com/forum/viewtopic.php?f=9&t=42540
User avatar
Tutor
Licensed Customer
Posts: 531
Joined: Tue Nov 20, 2012 2:57 pm
Location: Suburb of Birmingham, AL - Home of the Birmingham Civil Rights Institute

FrankPooleFloating wrote:
Tutor wrote: I don't have any particular case in mind that satisfies this first preference. However, secondly I'd then prefer a case with as much front (and top) grill space as possible to exhaust all, or as much as is possible, heat out of the front and top of the case...
Hey Tut, have you seen Thermaltake Core X9 and Core X5?...

http://www.thermaltake.com/Chassis/Cube ... design.htm

http://www.thermaltake.com/products-mod ... C_00002806

I'm going to finally pull the trigger and get a X9 in a couple weeks. And over the holidays do some upgrading...

X9 easily takes four hybrids and cpu aio, and even multiple 480mm rads in huge loops...

Thanks Frank,

The Thermaltake Core X9 looks to be a much better solution for those with hybrid water cooled (as well as solely air cooled) GPUs. Hats off to you for providing a stellar recommendation.
Because I have 180+ GPU processers in 16 tweaked/multiOS systems - Character limit prevents detailed stats.
Post Reply

Return to “Off Topic Forum”