Best Practices For Building A Multiple GPU System

[smicha]

Got all 7 running on one PSU and on a new mobo.

Also a x99e ws?

yes =)

yes yes yes ...3.1

[Notiusweb]

Smicha, if EVGA came out with some 3400W PSU (or whatever feeds 13), or if you would take 3 PSUs, can you visualize creating a 12-Gpu rendering + 1 display-piece, with 1x splitters in there?

[smicha]

Smicha, if EVGA came out with some 3400W PSU (or whatever feeds 13), or if you would take 3 PSUs, can you visualize creating a 12-Gpu rendering + 1 display-piece, with 1x splitters in there?

Notiusweb,

Not a problem - I am designing my custom case for fully watercooled X10DRX with 11 GPUS. The problem with the mobo is that only its lowest 8 pcie slots are open for 16x cards - 3 upper are for 8x due to collision with ram slots and cpus. So the solution here would be to use 16x to 8x extenders - the ones Tutor shared.

Do you mean 12 gpus on 12 cards or 6 titan Z+ 1card on x99 e ws? In such scenario (7 cards) I could use 900D with 480+3x240+ external mora. From my experience (2xZs on water) Zs are very hot and 2 of them draw 1350W (if I remember well from Piotr Adamczyk build). Such solution would require double the amount of rads I used (12 cooling units of 120mm). But the best choice is surely CaseLabs STH10 or even the newest THW10.

If you have any questions - please let me know.

[smicha]

BTW the 7x980ti on water clocks to 1530mhz on core (hasn't tested it for higher clocks) and draws up to 1550W (at no OC'ed at 1350mhz about 1370W). Score is close to 1000, temps are from 37 to 45C at 1000RPS of vardars.

[Seekerfinder]

Got all 7 running on one PSU and on a new mobo.

Also a x99e ws?

yes =)

Thanks Glimpse. Is it going to be your machine?
Seeker

[glimpse]

Thanks Glimpse. Is it going to be your machine?
Seeker

I wish so..but in this case =) I just helped to make it happen.

[Seekerfinder]

Thanks Glimpse. Is it going to be your machine?
Seeker

I wish so..but in this case =) I just helped to make it happen.

Ah. You were the PCIe riser cable in someone else's rendering. As we all know, PCIe risers are very important...

[Notiusweb]

Speaking of PCI Risers...

To use V3 without the crashing, I finally moved the 3 Zs on the Amfeltec cluster board to 3 USB risers and now have a total of 6 USB PCIE 3.0 risers working the 6 Zs, with my Titan X still in the Mobo.
I actually had been contemplating this for a while, but was dreading taking apart the cluster and the watercooling. But what I did instead was keep the cluster rig structure active, and moved out the board and replaced it with the USB risers.
So now no more crashing with V3, and everything works great. I really wonder what it was...I really wanted to have the Amfeltec work with V3 but for whatever reason, it just didn't work. Amfeltec tried but had no answers, I had no answers, and Otoy can only do so much.

Photo Apr 22, 9 32 34 AM.jpg

Photo Apr 22, 9 32 45 AM.jpg

So I learned a lesson about Best Practices, you can try to figure out a puzzle, but after a while the puzzle is just too much...you have to make a choice whether to spend more time trying to make something work, or just going with what works. I won't call it a divorce, but more a separation. They say when you're in a relationship, moving to V3 is one of the toughest challenges you face. I think we fell victim to that.
If I would be able to find what works with a fix and have it work, then I might go back and use it. Next rig I create I'll try with different parts and maybe it works. I really would like if Amfeltec and I could later still be friends,

On the flip side, now I can render like a Fender.

[itou31]

That's funny !, I went from USB riser to cluster for V3 ! So each hardware has it's "magic"

[glimpse]

So now no more crashing with V3, and everything works great. I really wonder what it was...I really wanted to have the Amfeltec work with V3 but for whatever reason, it just didn't work. Amfeltec tried but had no answers, I had no answers, and Otoy can only do so much.

would be cool to learn what is causing that..- sad to hear that wasted quite a bit of Your tiem to the point You decided to give up on that. These harware issues are soemtime so tricky to pinpoint & solve that is depressing to say the least.

& then..

That's funny !, I went from USB riser to cluster for V3 ! So each hardware has it's "magic"

=) fun times!

What leads me to believe.. the problem migh not be splitters or risers but soemthign else..in the system(s).

EDIT:

* as a matter of fact & reminder Amfeltecs backplane (again x1) seems to be working fine for eGPU build. Thus..1x might have nothing to do with crashes in the end.

** another idea, but I do not know how much ground it has..- Notius have You tried to ground multiple PSUs & those explansion units?