Best Practices For Building A Multiple GPU System

[smicha]

Hi Smicha,
Thanks for sharing your experience here. Which specific single PSU did you try (model & watts)? Sounds like an inrush issue and the spec / quality of the PSU would be critical here. What do you think? At 8 you said you tested the PSU but did try other units?
Seeker

The best available 1600W PSU - EVGA T2 supernova Titanium. Today I am receiving a new mobo and if needed I get an extra PSU. I'll send more feedback soon.

That's a great PSU. It could still be inrush though. Measuring at the wall might be useful.
The dual CPU / Xeon boards seem to be more stable with 5+ GPU's. Are you getting another x99e ws or a different board this time.
Cheers,
Seeker

Seeker,

The power draw for 4 980ti is 800W, so with 7 - approx 1300W. I must get same board - asrock ws e is out of stock, and no other board has 7 pci slots and fits in 900D. I wish x10drx would fit there or in CaseLabs. Hope to solve this shitty problem today. As for inrush - Stanley replied to me that even with 2000W I will not be able to run 6 or more gpus on a single power supply. This is crazy. Is there any reason Linus fired up 7 amd on a single 1600W?

[Seekerfinder]

My first Asus build handed out 00 codes so incessantly that I thought it had an affinity for James Bond, but just couldn't add a "7" after the double "0s".

Oh that's funny!

Tutor,
Do I understand correctly that you got that 00 error before loading the system with a bunch of rendering cards? I think Asus service worldwide seems to be a problem.
I also like the the Tyan boards.
Seeker

[smicha]

My first Asus build handed out 00 codes so incessantly that I thought it had an affinity for James Bond, but just couldn't add a "7" after the double "0s".

Oh that's funny!

Tutor,
Do I understand correctly that you got that 00 error before loading the system with a bunch of rendering cards? I think Asus service worldwide seems to be a problem.
I also like the the Tyan boards.
Seeker

This is exactly my case, Seeker.

[glimpse]

the funny thing.. is that code says nothing for ASUS as well =D

[Notiusweb]

One thing to look at too is what arrangement is the single PSU to be used in...as an entire supplier for everything, or just GPUs...

When I was getting GPU pre-watercooling, I looked at the actual connection ports on the PSU itself and I was able to successfully run 4 Titan Zs off each EVGA 1600 P2 & T2, they each take 2 of those 8-pin inputs. It has 9 dedicated to VGA and then it looks like you could borrow from the other sections as long as the plug fits in those little pin-hole slots.

EVGAT21600.jpg

Then after that the most PSU / GPU I was able to run off of one 15-amp breaker was 2x EVGA 1600W (P2 and T2) and a Thermaltake 1500W. So that would be my whole PC, 7 GPU, 6 Titan Z and 1 Titan X (in my case 13 cores) running off of 4700W, on a single 15-amp breaker. Included in that were all my disk drives, monitor, MOBO, etc. Because it was V2 in this scenario, I could only use 12GPU cores at a time for rendering. But I would use 12 GPU Titan Z as rendering and then the Titan X as the display, so they were all being 'used'. When I would run a quick render it would work, but then when I ran extended sessions or Octane Bench, it would trip the breaker. From that point I split it up on 2 separate breakers. So really you could always have an outlet strip to add as many PSUs as the breaker will allow.

I think it really matters more how many GPU could you run off of one PSU when you would want to have everything in one single case. Or if you only want the aesthetic of one single PSU in an external board arrangement.

[Tutor]

Tutor,

One question: did you RMA the 00 asus boards and was given a refund ?

Nope. I still have both of them as constant, life long reminders of how much I detest Asus products.

My first Asus build handed out 00 codes so incessantly that I thought it had an affinity for James Bond, but just couldn't add a "7" after the double "0s".

Oh that's funny!

Tutor,
Do I understand correctly that you got that 00 error before loading the system with a bunch of rendering cards? I think Asus service worldwide seems to be a problem.
I also like the the Tyan boards.
Seeker

Nope. I had installed OSes, drivers, etc. Then loaded up seven of my Titans (the original ones) on risers planted in the seven PCIe slots; then, that's when things went very bad with bios corruption. Between my Asus nightmare and my subsequent purchases of Tyan & Supermicro Servers, I ran the Titans successfully in one of my two old EVGA SR-2s systems (purchased back in 2010). They are 7 PCI-e slotted/ X16 sized - motherboards. There, the Titans ran without any incident. The SR-2s are now running GTX 590s. The bioses for the SR2s are known to be somewhat quirky; but what does that say about my Asus Servers' bios? I still smell their stench.

BTW - To power the Asus Servers and the EVGA SR-2 when running the seven Titans back in 2013, I used an EVGA 1600W PSU and an EVGA 800W PSU.

P.S. If one were to ask me what make of motherboards do I have presently that are still running? I'd say, Commodore (1980s-1990s), Atari (1980s-1990s), Aspen (1990s) , DeskStation (mid to late 1990s), Dell (early 2000s), Gateway (early 2000s), Apple (1980s-2010), Power Computing (mid to late 1990s), EVGA (2010 - 2014), Gigabyte (2000s to 2013), Tyan (2000s - 2013) and Supermicro (2013 - present); but if the question was what of make of motherboards do I still have presently that were pure crap from the get-go I'd have to say ("Asus" & "Asus") & "Asus" & "Asus".

[Notiusweb]

Just a shot in the dark, but maybe these posts help with the ASUS 00 code thing?:

http://www.tomshardware.com/answers/id- ... -code.html

http://www.tomshardware.com/answers/id- ... board.html

http://www.tomshardware.com/answers/id- ... -code.html

[smicha]

Got all 7 running on one PSU and on a new mobo.

[Seekerfinder]

Got all 7 running on one PSU and on a new mobo.

Also a x99e ws?

[glimpse]

Got all 7 running on one PSU and on a new mobo.

Also a x99e ws?

yes =)