Best Practices For Building A Multiple GPU System

[smicha]

The prices are crazy...

[Tutor]

The prices are crazy...

I agree, particularly because it appears from (a) the Cyclone manual, (b) the PSU supplied PCIe cabling, and (c) the chassis’ dimensions, to have been designed for 8 double-wide GPUs and, in that case, especially when compared to the price of this — viewtopic.php?f=40&t=43597&p=271307&hil ... ro#p271307 .

But, if (1) one already has a motherboard with, at least, 4 x16 sized slots and at least one of those slots is a true x16 slot always and in all aspects for the host card AND (2) the Cyclone backplane is in all use cases CPU and OS agnostic*/ even when connected to 16 double-wide GPUs on risers and supported by a self-made chassis like the bit-coin rig that I referenced earlier, then Cyclone’s prices might be seen as more reasonable because that would mean that one could run at least 19 GPUs IN WINDOWS. That might matter much when one's Octane and 3d application(s) run only in Windows. I’ll verify all of these points, especially the meanings/bottom lines of the paragraphs in the footnote, below, tomorrow.

*/ The manual states, in pertinent part:
(1) “the PCIe2-2710 Expansion System is recognized by the host system upon boot-up, requires no hardware specific drivers and is entirely host operating system agnostic.”
(2) “Once the PCIe-426 is installed into the host PC, the cable connected to the PCIe2-2710 chassis, the chassis plugged into an AC power outlet and any desired add-in cards are installed, the system is ready to be turned on. When the host is turned on, a signal from the PCIe-426 will turn on the PCIe2-2710 chassis. A number of things happen at this point. First, the PCI Express links are initialized. This is a purely hardware initialization where each PCI Express link is set up following a negotiation of lane widths by the two ends of each link. No firmware or operating system software is involved. Once the links are initialized or “trained”, there are LED indicators on each of the Cyclone Microsystems cards that indicate the links are trained. A detailed explanation of the LEDs follows later in this manual. [Emphasis added]
One essential requirement for system initialization is the ability of the host system’s BIOS to be able to enumerate the many bridges inherent in a complex PCI Express design. The links from the PCIe-426 to the PCIe-429 are created with PCI Express Switches. Each link looks like a PCI-to-PCI bridge to the Host’s BIOS. The number of bridges can add up quickly. Older BIOS may not have the resources to enumerate the number of bridges. Make sure that the BIOS on the host computer has the latest updated BIOS. If required, contact the host system’s manufacturer to make sure that the BIOS used can handle the large number of bridges that it will see in the system.”

[Notiusweb]

Tutor just reading this makes me realize that even if the peripheral can support the connectivity of so many devices, it's not guaranteed that it will work with your PC given that your BIOS is going to be the ultimate bouncer at the door of the bar. So the peripheral could be working 100% but if your BIOS says it's unable to support the device flow, as it were, it's all for not.

[Tutor]

Tutor just reading this makes me realize that even if the peripheral can support the connectivity of so many devices, it's not guaranteed that it will work with your PC given that your BIOS is going to be the ultimate bouncer at the door of the bar. So the peripheral could be working 100% but if your BIOS says it's unable to support the device flow, as it were, it's all for not.

NEVER FORGET EACH AND EVERY PREDICATE
Yes, even if the peripheral and your software license support the connectivity of 20 devices, having a fully working 20 GPU system is still a pipe dream unless additionally your hardware setup, your system's bios, your selected and application supported/required OS and your electrical power supply capability/strategy/consistency/implementation, all support a 20 GPU system. There were, and are still, a gang of bouncers at the bar's door. Don't be fooled by their being lined up in a row rather than being side-by-side. Just as you were required in the past, you're still now required (and shall be required in the foreseeable future) to body slam (or payoff) each and every bouncer individually, methodically and soundly before the sipping/gulping begins.

[smicha]

Tutor,

How many GPUs have you managed to run on X9DRX or any other Supermicro board?

[Tutor]

Tutor,

How many GPUs have you managed to run on X9DRX or any other Supermicro board?

I'm successfully managing 20 Keplers on the X9DRX under Linux OS. The potential gain hasn't yet been worth the potential loss of going further. Going beyond 20 would require that I chance disturbing the clean electrical power being supplied to other nearby 24/7 rendering systems and the extra heat I don't need. Because of ocean current born heating patterns (El Nino) it's hot as Hades here and summer hasn't even arrived.

[smicha]

Great. Could you list what cards are there on x9drx?

[Tutor]

Great. Could you list what cards are there on x9drx?

Yes:

14 GPU Processors - All fully water cooled - 6 Titan Zs, 1 Titan, 1 Titan Black and

6 GPU Processors - 6 x GTX 780 6Gs - Air - now; but I have also used successfully 6 x GTX 780 TI - Hybrids and 6 x GTX Titan - Air.

P.S.

1) For ease of transport/swapping, as well as stability/mounting, I keep groups of many of my GPUs, on home-made racks like this - download/file.php?id=42856&mode=view .

2) Because I was (and still am somewhat) a Mac addict, I prefer to do much of my pre-final render duties on my pre-2013, i.e., classic MacPros, so I'll soon be modifying my classic MacPros to support external GPUs like this - download/file.php?id=41856&mode=view .

[smicha]

One more question - do all of them work as 4x?

[Tutor]

Things Change And My Tech Life Is Full Of Compromises And Thus Are My Systems Continuously Undergoing Change and Are Reflective Of Compromises

One more question - do all of them work as 4x?

Short Answer To Your Question: No. Three devices work as x4, eight devices work as x8 and six devices work as x1. I know that's more than 11 devices ( in fact it's 17) , but the devil is in the details. So, say "Hello" to Hell.

Problem1: The X9DRX has only eleven x8 (sized) PCIe slots. Slot 11 operates at x4 V2. The other slots operate at x8 V3.*/
Problem2: Windows appears to support only 13-14 GPU processors vs. Linux which supports many more.
Given: I run a number of OSes on my systems, including Windows, Linux Mint and FruityPHI (It's one of my works always in progress).

1) Non-GPU Card
In Slot 11 (otherwise blocked by motherboard components) (and farthest to the right - looking at the motherboard from front-to-back) is x4 card (in an x8 sized slot) that allows me to connect externally to a small raid case presently containing 4 x hard drives.

2) Amfeltec GPU Oriented Splitters
In Slots 10 and 9 (both otherwise blocked by motherboard components) are x4 Amfeltec GPU Oriented Splitters (in x8 V3 slots). 3 GTX 780s are connected to each splitter card (so it's x1 V2 at the GPU level). Having them connected to the splitter cards facilitates the quick change artistry in me.

3) x8 V3 Connected GPUs - All Titans = 6 x Titan Z + 1 x Titan Classic (the first Titan) + 1 x Titan Black
In Slot 8 (otherwise blocked by motherboard components) is an x8 to x16 riser card (not cable) [ download/file.php?id=43956&mode=view ] to which is attached an x16 to x16 riser cable leading to a Titan Black (so it's x8 V3).
In Slot 7 is an x8 to x16 riser card (not cable) to which is attached a Titan Z[#1] (so it's x8 V3).
In Slot 6 is an x8 to x16 riser card (not cable) to which is attached an x16 to x16 riser cable leading to a Titan (so it's x8 V3).
In Slot 5 is an x8 to x16 riser card (not cable) to which is attached a Titan Z[#2] (so it's x8 V3).
In Slot 4 is an x8 to x16 riser card (not cable) to which is attached an x16 to x16 riser cable leading to a Titan Z[#3] (so it's x8 V3).
In Slot 3 is an x8 to x16 riser card (not cable) to which is attached a Titan Z[#4] (so it's x8 V3).
In Slot 2 is an x8 to x16 riser card (not cable) to which is attached an x16 to x16 riser cable leading to a Titan Z[#5] (so it's x8 V3).
In Slot 1 is an x8 to x16 riser card (not cable) to which is attached a Titan Z[#6] (so it's x8 V3).
When this systems is next taken off-line, I may rid it of the x8 to x16 riser cards by slitting the backs of many of the x8 slots on the motherboard, namely the ones that aren't blocked by motherboard components) so that I can then place either the riser cables and the cards themselves directly into those motherboard PCIe x8 slots IF I determine that I can still insert safely and maintain safely the x16 to x16 riser cables in the slots immediately adjacent to the video cards. However, in the event that I have to keep the video cards on the x8 to x16 riser cards to have the space for the splitters, I could slit the backs of only the slots for the splitters to place the splitters directly in those PCIe slots..

I now firmly believe that the case should take a backseat to the internals and be designed only after optimal placement of internals and their performance have been maximized. Thus, the next time that I build a 20+ GPU multiple GPU system, I'll start by first designing and then building a metal framework like this - https://www.youtube.com/watch?v=BSn3FC_SBPA - for the mother board, GPUs, PSUs, storage, etc., but also with strong handles to move the whole framework. Then, only after I had fully tested the build and found it to be completely satisfactory would I then consider how to best encase it, including providing for maximum cooling.

*/
(1) A V3 slot can operate at twice the speed of a V2 slot.
(2) Not too long ago I would have thought that I'd never say that having only 11 PCIe slots posed a challenge, but things do change.

Hell's over. You're free to leave; but, if you like it here, there's an extended stay option - it's call "Eternity."