Best Practices For Building A Multiple GPU System

[Tutor]

Tutor, on the topic of > 12 GPU, where does the Thea and FurryBall limit sit (or is there a limit)? Also, thoughts on speed (vs Octane if it serves to simplify a description)?
BTW, just thoughts, did you ever see Indigo Render, it is nice but only allows 1 GPU as CUDA at the present time. NVidia Iray, at least for Daz Studio, allows 13 GPU, but is slow. Bunkspeed Zoom looks like it is good at doing cars, but who knows about the rest (animations, characters, effects, etc...)

1. Point of View Matters
I call all of the GPU renders that I use beta because they're always playing catchup with the underlying 3d packages. Also, each of them offers one or features to try to distinguish it from competitors. For example, Thea can harness the render prowess of your system's GPUs and CPUs. And we must not forget the quirks, e.g., FurryBall states: "When you combine differents GPUs for rendering, fast GPU will wait for slower one and it will cause the slower rendering."

2. GPU Usage And Scaling
Although neither TheaRender nor FurryBall has a licensed system GPU limit like Octane does, TheaRender scales the best for large numbers of GPUs per system for my uses. FurryBall states: "Number of GPU in single computer is unlimited, but we recommend 3-4 at maximum, because more GPU is not so effective and over head is too big." Redshift 3d, which you didn't mention, allows one to run multiple instances on the same system (splitting the render job) when one has more than 8 GPUs - an instance can run up to eight GPUs currently. But one can run multiple instance of Redshift 3d on the same system under a single license. The recently released FurryBall RT appears to be a better scaler than the up-to-date render version then being sold when the above-stated caution by FurryBall about 3-4 GPUs at maximum was first issued. Redshift, TheaRender and Furryball appear to me to be as fast as, if not faster than, Octane for certain scenes (but in the case of FurryBall that would be comparing 4 GPUs to 4 GPUs in Octane so it not comparing 12 apples to 12 apples). For full disclosure, I'm running MacPros and mainly self-built systems with Windows, Linux and MacOS installed on them.

3. The More The Merrier
Moreover, all of these different renderers have their own material/texture differences, some just small differences and others not so small, as well as what I'd call fillers for feature gaps of one or more of its competitors. For example, Redshift3d gets around the low GPU memory amounts in my GTX 590s and 480s by means of an excellent bucket renderer for large formats. Also, I tend to use at least two GPU renders together (Yes! You can do that) in the early stages of project development to get the precise (or closest) look that I'm after in the end. As to the others renderers named I've tried some of them out, but I'd recommend using TheaRender, FurryBall, Octane, Redshift or Blender Cycles ( and if you can afford it and it pertains to your underlying 3d application and project - a combination of them).

4. Decide For Your Self/System And For Your Particular Use Cases
1) http://furryball.aaa-studio.eu/products ... rsion.html
2) https://www.redshift3d.com/demo
3) https://www.thearender.com/site/index.php/downloads
4) https://www.blender.org/download/

[smicha]

Thanks Tutor for great as usual thoughts.

[Notiusweb]

With talk of 20 GPU limit coming soon in the V3 release forums, we are going to have to find more breaker outlets.
Extension chords ahoy!

[Seekerfinder]

With talk of 20 GPU limit coming soon in the V3 release forums, we are going to have to find more breaker outlets.
Extension chords ahoy!

I long for the day when breakers are the limit, not Octane's GPU count! Looks like it may finally happen.
Seeker

[Tutor]

With talk of 20 GPU limit coming soon in the V3 release forums, we are going to have to find more breaker outlets.
Extension chords ahoy!

In order of my preference:
1) I hope for no GPU cap per system license, just as Octane's competitors provide.
2) But, a cap no lower than 24 GPUs per system license would be welcomed.
3) However, a limit of 20 GPUs per system license would be a lot better than 12 GPUs per system license.
4) At the very least, for those with more than one Octane license, that the 12 GPU cap be applied only to the license limit and be henceforth disassociated with the system GPU limit. Extra licenses could be used additively for a system so that a user with multiple licenses can use each of those licenses, in excess of one, to add up to twelve more GPUs to a system for each unused license the user has in excess of one. For example, if a user has 3 licenses, the user can run up to 36 GPUs in Octane in a single system or use all 36 GPUs via Octane Network Rendering. That policy would promote the sale of more licenses => providing more income to Otoy, while giving the customer greater flexibility in how each license is deployed.

[Seekerfinder]

With talk of 20 GPU limit coming soon in the V3 release forums, we are going to have to find more breaker outlets.
Extension chords ahoy!

In order of my preference:
1) I hope for no GPU cap per system license, just as Octane's competitors provide.
2) But, a cap no lower than 24 GPUs per system license would be welcomed.
3) However, a limit of 20 GPUs per system license would be a lot better than 12 GPUs per system license.
4) At the very least, for those with more than one Octane license, that the 12 GPU cap be applied only to the license limit and be henceforth disassociated with the system GPU limit. Extra licenses could be used additively for a system so that a user with multiple licenses can use each of those licenses, in excess of one, to add up to twelve more GPUs to a system for each unused license the user has in excess of one. For example, if a user has 3 licenses, the user can run up to 36 GPUs in Octane in a single system or use all 36 GPUs via Octane Network Rendering. That policy would promote the sale of more licenses => providing more income to Otoy, while giving the customer greater flexibility in how each license is deployed.

Spot on Tutor.

Seeker

[glimpse]

Agree with this..because now OTOY shoots at their foot with curent limmit - there are small studios & even freelancers building multiple rigs with 4-8GPUs each & even if they have licences ..they can not use them for the same render work..that's crazy..

I mean, ignoring comunity might lead to completely different outcome..but heck, I believe OTOY has some sort of marketing group that makes sense of all those decissions.. (let's keep hope v3 is going to lift some of those unreasonable limmits).

[Proxell]

With talk of 20 GPU limit coming soon in the V3 release forums, we are going to have to find more breaker outlets.
Extension chords ahoy!

In order of my preference:
1) I hope for no GPU cap per system license, just as Octane's competitors provide.
2) But, a cap no lower than 24 GPUs per system license would be welcomed.
3) However, a limit of 20 GPUs per system license would be a lot better than 12 GPUs per system license.
4) At the very least, for those with more than one Octane license, that the 12 GPU cap be applied only to the license limit and be henceforth disassociated with the system GPU limit. Extra licenses could be used additively for a system so that a user with multiple licenses can use each of those licenses, in excess of one, to add up to twelve more GPUs to a system for each unused license the user has in excess of one. For example, if a user has 3 licenses, the user can run up to 36 GPUs in Octane in a single system or use all 36 GPUs via Octane Network Rendering. That policy would promote the sale of more licenses => providing more income to Otoy, while giving the customer greater flexibility in how each license is deployed.

Hey Tutor,

We are trying to build 8 gpu system based on titans. Can you help us? We have some questions:

1.Do we need a standalong card for windows interface handle? Does it solve interface lags?
2.Do we need tcc mode for other calc-cards? Are there any benefits from tcc mode in means of render time?
3.Tcc mode is only available for cards that have a similar chip with Tesla, or any titan can be switched to tcc mode?

Thx

[Tutor]

With talk of 20 GPU limit coming soon in the V3 release forums, we are going to have to find more breaker outlets.
Extension chords ahoy!

In order of my preference:
1) I hope for no GPU cap per system license, just as Octane's competitors provide.
2) But, a cap no lower than 24 GPUs per system license would be welcomed.
3) However, a limit of 20 GPUs per system license would be a lot better than 12 GPUs per system license.
4) At the very least, for those with more than one Octane license, that the 12 GPU cap be applied only to the license limit and be henceforth disassociated with the system GPU limit. Extra licenses could be used additively for a system so that a user with multiple licenses can use each of those licenses, in excess of one, to add up to twelve more GPUs to a system for each unused license the user has in excess of one. For example, if a user has 3 licenses, the user can run up to 36 GPUs in Octane in a single system or use all 36 GPUs via Octane Network Rendering. That policy would promote the sale of more licenses => providing more income to Otoy, while giving the customer greater flexibility in how each license is deployed.

Hey Tutor,

We are trying to build 8 gpu system based on titans. Can you help us? We have some questions:

1.Do we need a standalong card for windows interface handle? Does it solve interface lags?
2.Do we need tcc mode for other calc-cards? Are there any benefits from tcc mode in means of render time?
3.Tcc mode is only available for cards that have a similar chip with Tesla, or any titan can be switched to tcc mode?

Thx

1. A standalone card does facilitate Windows interface handling, eliminating interface lags. I use GT 640 4Gs. I've had them for a few years. There're newer, cheap 4G, low powered, single slot cards that support 4k now and the Maxwells support 4k over HDMI 2.0. 4k TVs are now cheaper, but with larger displays, than current 4k monitors, but unlike the monitors, the TVs have only HDMI connectors (many of them have HDMI 2.0 for 60Hz 4k), but the TVs do not have displayport connectors.
2. TCC mode isn't needed for other GTX calc-cards. It can help reduce IO space usage and thus facilitate placement of more working cards in a system running up against its IO space limits. It might benefit render times depending on the particular rendering engine. TCC mode tends to increase only floating point operations. So the speed benefit to Octane users is dubious.
3. TCC mode is not only driver-triggered, but it's benefits are driver-based. Thus, while TCC mode can be triggered on most GTX GPUs, the other features of triggering it are mainly reserved for true Teslas or GTX cards that share the same or very similar GPU type as Tesla cards. "Yes, any Titan, using the appropriate driver, and mod [ http://http.developer.nvidia.com/Parall ... luster.htm ] can be switched to TCC mode." But for Octane speed-related purposes - there'll be little to no performance gain [ https://devtalk.nvidia.com/default/topi ... /?offset=2 ]. On/In Octane, the best GTX cards smoke the best Teslas.

Conclusion: I'd use the TCC mode to help reduce IO space requirements for video display related card functions if one of my systems ran up against its IO space limits for the number of GPUs I wanted to install or if I was using a renderer that relied significantly on floating point operations.

[Tutor]

With talk of 20 GPU limit coming soon in the V3 release forums, we are going to have to find more breaker outlets.
Extension chords ahoy!

Happy every days to all.

NoticeWeb,

So true are your power related comments.


Here's that update that I promised to give:

System no. 1 Goal = Have four PSUs, motherboard, lots of storage, et., and 20+ water touched GPU processors all in one case.

My first Lian LI [ LIAN LI PC-D8000 Black Aluminum ATX Full Tower Computer Case - bought here: http://www.newegg.com/Product/Product.a ... 6811112390 for about $345 w/shipping ] test case has fallen on it's side and I may not let it get upright again (I may just remove the wheels). Reason = so that (1) my Supermicro X9DRX+-F motherboard (hereinafter "MOBO'), w/2 x CPUs=2xE5-4650 ES QBED = 2xE5-2680 V1s for 16-cores w/o counting hyper-threading, will be oriented at the bottom of the case which is the coolest place in the case, and (2) my GPUs, hanging from the rail that I made, will best remain properly spaced so that the GPUs cool better.

Here're some pics of what I'm toying with doing to the first of my five Lian Li encased multiple GPU systems. Currently, I've installed six of my air-cooled original GTX Titans in there in a two story stacking fashion - three Titans (system attached by Amfeltech GPU oriented splitters, but I've been considering seating those three elevated Titans on x16 to x16 riser cables which would themselves be seated in x8 to x16 slot riser cards */ ). Those three GPUs sit above three more Titans (attached to PCIe slots via x8 to x16 slot risers cards (not riser cables). Aside: I intend to eventually convert all of my original GTX Titans to hybrids. But first, after setting up all of my software, I want to begin incrementally adding my fully water-cooled (A) 6 x GTX Titan Zs, (B) 1 x GTX Titan Black and (C) 1 x GTX Titan (the original one - now fully water-cooled) to work out all IO space issues.

I've also installed 5 x Western Digital HDs (leaving me room for installing only an additional 12-13 (size dependent) more of them because of the storage space now occupied by the 2 x 1,600W EVGA PSUs I've installed in the front of the case). I've also installed (A) on the MOBO 128Gs of ram, (B) in the case - a Plextor DVD drive, and (C) on the motherboard I've also installed (1) 3 x Amfeltec GPU Oriented Splitters, (2) a PCIe-based SATA card for connecting my four-drive external storage chassis to back up files and (3) a OCZ Storage Solutions RevoDrive 350 Series 960GB PCI Express Generation 2 x 8 Solid State Drive which supports up to 1800MB/s reads, 1700MB/s writes, and 140,000 random write IOPS.


Powering the beasts:
(A) 4 x 1600W PSUs (2 fed power from rear of case and 2 fed power from front of case = 1 x LEPA [ultimately intended to power the MOBO, 2 x GTX Titans (<700W) and other MOBO system power needs] and 3 x EVGAs [powering 6 x GTX Titan Zs (<6x550W=<3,300W), 3 x GTX Titans and one Titan Black (<4x350W = <1400W)] and
(B) 1 x 450W (500W peaking capacity) BoosterX5 PSU for powering 2 of the 3 x GTX Titans that are PCIe fed 75W from their (essentially) direct connection to the MOBO.


*/ N.B.
(1) All eleven PCIe slots on the Supermicro X9DRX+-F's motherboard are x8 size (the eleventh one is x4 signal wise, but x8 in size and its where I placed my PCIe-based SATA card ). I'm planning on using only eight of the motherboard's eleven slots immediately, leaving me with three free slots for placing Pascals. Those additional GPUs may require my making additional modifications, and even consider out of case placement.
(2) Pic related info (note that in all of the pics chassis is resting on it's left side):
(a) First pic (top-most one) shows a view of what would be normally the top of the Lian Li chassis. Those fans are oriented to suck air into the case because the case won't be sitting upright. In fact, there're air intakes on what will functionally, for me, be the bottom of the case (but it's really what would be the left side cover of case if I were to set the case upright) and there will be fans w/air filtering there, sucking cool air upwards into the case and another set of fans to pull cooler air up into the case to better cool the PSUs installed in the front of the case) and the air intakes and filters on the cover for the top of the case (but it's really what would be the cover for the right side cover of case if I set the case upright) will be used to exhaust warmer air. I'm aiming for creating a lot of intake air pressure to keep the whole system as cool as is possible.
(b) Second pic shows what would normally be a view of the right side of the chassis.
(c) Third pic shows what would normally be the front of the chassis, sitting upright. Of course, I'll be modifying what would normally be the case's front cover to secure and provide an exhaust route for the two PSUs there.
(d) Like the second pic, the fourth and fifth pics show what would normally be views of the right side of the chassis, but from different perspectives to assist in gaining depth perspective so that one can get a better idea of the GPU stacking that I have going on. To nail it - compare my pics with pics here: http://www.lian-li.com/en/dt_portfolio/pc-d8000/ .