Best Practices For Building A Multiple GPU System

[Notiusweb]

So I'm using other GPU renderers for the time being because for my current projects I need support for, at a minimum, sixteen GPUs for a number of my systems.

Very nice!
And never forget the famous GPU-rig reference from Sound of Music..."I am 16, going on 17..."

[Notiusweb]

Smicha, I looked at your pics of that latest build and it feels like a movie trailer, or like a C4D effects reel or something. Like you're in a dark futuristic science lab.
And the ones with the viola/violin are awesome....I would say next time get a bottle of wine for the background, and also some rose petals fluttered at the base of the machine, to enhance the mystique and elegance. I'm not trying to be silly, it looks artistic, that's why I am saying that. I like that you used the EVGA 1600, it gives you all you could need.

[smicha]

Smicha, I looked at your pics of that latest build and it feels like a movie trailer, or like a C4D effects reel or something. Like you're in a dark futuristic science lab.
And the ones with the viola/violin are awesome....I would say next time get a bottle of wine for the background, and also some rose petals fluttered at the base of the machine, to enhance the mystique and elegance. I'm not trying to be silly, it looks artistic, that's why I am saying that. I like that you used the EVGA 1600, it gives you all you could need.

Thank you.

[Notiusweb]

Tutor, it just dawned on me to ask you a question, which I believe I never have done before...
Is that Supermicro board you have (X9DRX or X10DRX) capable of running 16GPUs? As in, you can boot into your OS with 16 GPUs and then run software allowing full GPU usage?
I guess what I am wondering is when you refer to 16, is that on a single system, or is that a combo of single systems networked together...

[Tutor]

Tutor, it just dawned on me to ask you a question, which I believe I never have done before...
Is that Supermicro board you have (X9DRX or X10DRX) capable of running 16GPUs? As in, you can boot into your OS with 16 GPUs and then run software allowing full GPU usage?
I guess what I am wondering is when you refer to 16, is that on a single system, or is that a combo of single systems networked together...

My three X9DRXs are each capable of running, at least, 20+ GPU processors (for instance - either 20+ single GPU processor Titans or 10+ dual GPU processor Titan Zs or a combination thereof equal to 20+ GPU processors) with Linux, or up to 13 [or 14 GPU processors if the 14th GPU is a GT 640 4G] with Windows. I run multiple OSes, at least three, on all of my systems. I settled upon a current max of 16 GPUs so I can add Pascals and Voltas to each of those systems later and hopefully Octane will, like its competitors, no longer impose that 12 GPU processor per licensed system max.

[Notiusweb]

Tutor, sorry to ask this, you probably mentioned before. With that Amfeltec 4-way GPU oriented splitter, did you have it going straight to the card from the board at 4x, or is there a riser connection in the mix with that piece making it 1x.

Also, which Otoy Smiley do you like the best:

[Tutor]

Tutor, sorry to ask this, you probably mentioned before. With that Amfeltec 4-way GPU oriented splitter, did you have it going straight to the card from the board at 4x, or is there a riser connection in the mix with that piece making it 1x.

The splitters sit in 8x slots, while the GPUs that aren't connected by splitters sit on PCI Express Riser Cards x8 to x16 Left Slot Adapter For 1U Servers which I will have purchased [ viewtopic.php?f=40&t=43597&start=190#p238446 ].

Also, which Otoy Smiley do you like the best:

I'm not really into Smileys, but if I had to chose one it would be , considering that I'm an old geek.

[Notiusweb]

The splitters sit in 8x slots, while the GPUs that aren't connected by splitters sit on PCI Express Riser Cards x8 to x16 Left Slot Adapter For 1U Servers which I will have purchased [ viewtopic.php?f=40&t=43597&start=190#p238446 ].

Thanks Tutor.

***Bonus Question***
(a) If Tutor has the Amfeltec 4-way GPU Splitter going to mobo,
(b) and that splitter connects to the mobo with a 4x male connector into his board's 8x female connector,
(c) lets say using a total of 4 GPU on the splitter),

that means that each card potentially runs at a max bandwidth of:
(1) PCIE 4x (each GPU can run simultaneously at the max connection size of 4x), or
(2) PCIE 1x (each GPU runs at 4x/# GPU, or 4/4 = 1)

Car-rash.png

[Tutor]

The splitters sit in 8x slots, while the GPUs that aren't connected by splitters sit on PCI Express Riser Cards x8 to x16 Left Slot Adapter For 1U Servers which I will have purchased [ viewtopic.php?f=40&t=43597&start=190#p238446 ].

Thanks Tutor.

***Bonus Question***
(a) If Tutor has the Amfeltec 4-way GPU Splitter going to mobo,
(b) and that splitter connects to the mobo with a 4x male connector into his board's 8x female connector,
(c) lets say using a total of 4 GPU on the splitter),

that means that each card potentially runs at a max bandwidth of:
(1) PCIE 4x (each GPU can run simultaneously at the max connection size of 4x), or
(2) PCIE 1x (each GPU runs at 4x/# GPU, or 4/4 = 1)

Car-rash.png

When I previously guessed that the newer Amfeltec GPU-Oriented x1 PCIe 3-Way Splitter [ http://amfeltec.com/products/flexible-x ... -oriented/ ] differs from the older Amfeltec GPU-Oriented x4 PCIe 4-Way Splitter [ http://amfeltec.com/products/flexible-x ... -oriented/ ] only by the number of ports for GPU card connectors (3-way vs. 4-way), I didn't point out that the newer Amfeltec GPU-Oriented x1 PCIe 3-Way Splitter is, of course, an x1 card, unlike the Amfeltec GPU-Oriented x4 PCIe 4-Way Splitter which is an x4 card (and there might be other chip differences). So if response no. 2, above, is correct as to the x4 Splitter, then if the question had been about the 3x Splitter, then the answer would most likely be that each card on the 3-way Splitter would run at a max bandwidth of PCI-e x0.333333... (each GPU runs at x1/# GPU, or 1/3 = x0.333333... or in any event less than x1 when running three GPUs) . But if response no. 1, above, is correct as to the x4 Splitters, then if the question had been about the 3x Splitter, then the answer would most likely be that each card on the 3-way Splitter would run at a max bandwidth of PCI-e x1 (each GPU can run simultaneously at the max connection size of x1. Rereading the manuals gives me the impression that response no. 2 might be correct, but the manuals are inartfully worded and the one Amfeltec indicates is for my cards loosely refers to a number of different card configurations and none of the pictures in the manual for Amfeltec GPU-Oriented x4 PCIe 4-Way Splitter depicted there actually depict the Amfeltec GPU-Oriented x4 PCIe 4-Way Splitter cards that Amfeltec sent to me, i.e., the manuals are down-right confusing and under-inclusive.

"Lane[edit]
A lane is composed of two differential signaling pairs, with one pair for receiving data and the other for transmitting. Thus, each lane is composed of four wires or signal traces. Conceptually, each lane is used as a full-duplex byte stream, transporting data packets in eight-bit "byte" format simultaneously in both directions between endpoints of a link.[7] Physical PCI Express links may contain from one to 32 lanes, more precisely 1, 2, 4, 8, 12, 16 or 32 lanes.[6][4]:4,5 Lane counts are written with an "×" prefix (for example, "×8" represents an eight-lane card or slot), with ×16 being the largest size in common use.[8]" https://en.wikipedia.org/wiki/PCI_Express [ Emphasis added].

Each flex cable from each of my x4 PCI Express Host cards to each of four x16 PCI Express adapter boards per card, seems to have, at least, 15 wires and signal traces.

In light of all of the above, I'm left wondering which, if either, scenario you posit is most likely true. However, my GPUs work to my satisfaction.

[Notiusweb]

The splitters sit in 8x slots, while the GPUs that aren't connected by splitters sit on PCI Express Riser Cards x8 to x16 Left Slot Adapter For 1U Servers which I will have purchased [ viewtopic.php?f=40&t=43597&start=190#p238446 ].

Thanks Tutor.

***Bonus Question***
(a) If Tutor has the Amfeltec 4-way GPU Splitter going to mobo,
(b) and that splitter connects to the mobo with a 4x male connector into his board's 8x female connector,
(c) lets say using a total of 4 GPU on the splitter),

that means that each card potentially runs at a max bandwidth of:
(1) PCIE 4x (each GPU can run simultaneously at the max connection size of 4x), or
(2) PCIE 1x (each GPU runs at 4x/# GPU, or 4/4 = 1)

Car-rash.png

When I previously guessed that the newer Amfeltec GPU-Oriented x1 PCIe 3-Way Splitter [ http://amfeltec.com/products/flexible-x ... -oriented/ ] differs from the older Amfeltec GPU-Oriented x4 PCIe 4-Way Splitter [ http://amfeltec.com/products/flexible-x ... -oriented/ ] only by the number of ports for GPU card connectors (3-way vs. 4-way), I didn't point out that the newer Amfeltec GPU-Oriented x1 PCIe 3-Way Splitter is, of course, an x1 card, unlike the Amfeltec GPU-Oriented x4 PCIe 4-Way Splitter which is an x4 card (and there might be other chip differences). So if response no. 2, above, is correct as to the x4 Splitter, then if the question had been about the 3x Splitter, then the answer would most likely be that each card on the 3-way Splitter would run at a max bandwidth of PCI-e x0.333333... (each GPU runs at x1/# GPU, or 1/3 = x0.333333... or in any event less than x1 when running three GPUs) . But if response no. 1, above, is correct as to the x4 Splitters, then if the question had been about the 3x Splitter, then the answer would most likely be that each card on the 3-way Splitter would run at a max bandwidth of PCI-e x1 (each GPU can run simultaneously at the max connection size of x1. Rereading the manuals gives me the impression that response no. 2 might be correct, but the manuals are inartfully worded and the one Amfeltec indicates is for my cards loosely refers to a number of different card configurations and none of the pictures in the manual for Amfeltec GPU-Oriented x4 PCIe 4-Way Splitter depicted there actually depict the Amfeltec GPU-Oriented x4 PCIe 4-Way Splitter cards that Amfeltec sent to me, i.e., the manuals are down-right confusing and under-inclusive.

"Lane[edit]
A lane is composed of two differential signaling pairs, with one pair for receiving data and the other for transmitting. Thus, each lane is composed of four wires or signal traces. Conceptually, each lane is used as a full-duplex byte stream, transporting data packets in eight-bit "byte" format simultaneously in both directions between endpoints of a link.[7] Physical PCI Express links may contain from one to 32 lanes, more precisely 1, 2, 4, 8, 12, 16 or 32 lanes.[6][4]:4,5 Lane counts are written with an "×" prefix (for example, "×8" represents an eight-lane card or slot), with ×16 being the largest size in common use.[8]" https://en.wikipedia.org/wiki/PCI_Express [ Emphasis added].

Each flex cable from each of my x4 PCI Express Host cards to each of four x16 PCI Express adapter boards per card, seems to have, at least, 15 wires and signal traces.

In light of all of the above, I'm left wondering which, if either, scenario you posit is most likely true.

Yikes..this means in the case of a Titan Z (dual core) on a 4-way cluster, with 3 Titan Zs like I have, =6 cores on a 1x connection,
Each GPU might only be 0.16 eligible (1/6). And that is if the board even works that way... It might already, by default, parse out its 4 GPU inputs to the 1x in .25 increments, in which case a Titan Z core/GPU would only be PCI 0.125x eligible.
No wonder why it is crashing, it can't handle the fury of V3's Kernels. Maybe if I had 4 regular single core titans it wouldn't crash.

But then why does the USB riser cause an issue for some users, and then not for me. Is it the make of theriser...like what does that mean....it should either connect at a certain # or not, unless in reality there is a gradient connection at play, where there is a potential max and min, with a grey area in between.

And of course this is all hypothetical, but it's the kind of hypothetical I like.

However, my GPUs work to my satisfaction.