Best Practices For Building A Multiple GPU System

[smicha]

I have no knowledge how the power draw is distributed - whether this is via a pcie slot or power connectors.

The mobo supplies 75w through each PCIe slot. The rest is auxiliary power straight from the PSU.

Seeker

Thank you for this, Seeker.

[Tutor]

I have no knowledge how the power draw is distributed - whether this is via a pcie slot or power connectors.

The mobo supplies 75w through each PCIe slot. The rest is auxiliary power straight from the PSU.

Seeker

Seeker,
By your response, are you saying that if I were to run an application that draws more than 375 watts (standard is 75 watts from the PCIe slot on the motherboard plus 150 watts from each of two 8-pin PCIe power cable connections [ 150 watts a piece ], say 500 watts, that the excess 125 watts comes only through the two 8-pin PCIe power connection cables and that absolutely none of the excess comes through the PCIe slot on the motherboard? The accuracy of your response is critical to what and how many power supplies I use, how I connect the PCIe cables to which power supply, what I leave powered by the motherboard, etc.

[Tutor]

Hi, Tutor.

I knew You were crazy (in a best possible way =)

anyway, happy to help with anything & happy to be in the same boat (forum),
as we all learn from each other here - more discussion the better is outcome!

Looking forward for Your build! & fingers crosse Your idea would work-out!
Insane build & there'll be mind-blowing numbers!

tom

I'm crazy is every way possible + I'm precision nutty - that's the mathematician in me.

[smicha]

Tutor,

If this helps in any aspect: my system at idle (i7 2600k @1.3V @4.5ghz, 2xD5 pumps, 2x HDD, 1xSSD, 16 fans) draws 180W.

[Seekerfinder]

I have no knowledge how the power draw is distributed - whether this is via a pcie slot or power connectors.

The mobo supplies 75w through each PCIe slot. The rest is auxiliary power straight from the PSU.

Seeker

Seeker,
By your response, are you saying that if I were to run an application that draws more than 375 watts (standard is 75 watts from the PCIe slot on the motherboard plus 150 watts from each of two 8-pin PCIe power cable connections [ 150 watts a piece ], say 500 watts, that the excess 125 watts comes only through the two 8-pin PCIe power connection cables and that absolutely none of the excess comes through the PCIe slot on the motherboard? The accuracy of your response is critical to what and how many power supplies I use, how I connect the PCIe cables to which power supply, what I leave powered by the motherboard, etc.

Tutor,
I don't know enough about motherboard power management to definitively answer that question, however, this is my understanding, yes. The motherboard will likely not allow more than 75W through the PCIe port. If your card tries to draw more power than the available amps you're likely to get instability.

I should add that I am not sure how far above a card's specified TPD power would be allowed by the manufacturer on a continuous basis. Power management on modern PSU's as well as more intelligent power management from card manufacturer's has lead to fewer blue screens in recent years. I guess you can hack anything but I'm not sure if any card will be able to use 500W dedicated power!

Here is an extract from Wikipedia on PCIe:
"... All sizes of ×16 cards are initially 25 W; like ×1 cards, half-height cards are limited to this number while full-height cards may increase their power after configuration. They can use up to 75 W (3.3 V × 3 A + 12 V × 5.5 A), though the specification demands that the higher-power configuration be used for graphics cards only, while cards of other purposes are to remain at 25 W.

Optional connectors add 75 W (6-pin) or 150 W (8-pin) power for up to 300 W total (2×75 W + 1×150 W). Some cards are using two 8-pin connectors, but this has not been standardized yet, therefore such cards must not carry the official PCI Express logo. This configuration would allow 375 W total (1×75 W + 2×150 W) and will likely be standardized by PCI-SIG with the PCI Express 4.0 standard. The 8-pin PCI Express connector could be mistaken with the EPS12V connector, which is mainly used for powering SMP and multi-core systems."
(page link: http://en.wikipedia.org/wiki/PCI_Express

You'll have to experiment and let us know how it goes!

Seeker

[Tutor]

Then The Titan Z Is The Real Deal.

Tutor,

I have no knowledge how the power draw is distributed - whether this is via a pcie slot or power connectors. The (#gpus+1)*300W rule allows me for rough approximation of PSU nominal power I shall get to run certain number of gpus safely. For example, for 7 Titans Z I would go with 3x1500W PSUs (15x300W), although entire system while rendering in Octane may draw about 3000-3300W (14x200W for gpus + 200-500W for the rest). So with 3x1500W PSUs there should be plenty of safety/stability reserves (even for heavy overclocking).

PS. Sorry if I missed it - if there is a limit of 12 gpus in Octane how will you make use of 7 Zs? Will you render on 6 and excessive 1 will handle the system?

I don't know either how the excess power above the TDP is distributed. I do know that in a perfect world that when a PCIe card has two 8-pin PCIe connectors that those are there to allow the card to draw at least 375 watts (75 watts from the slot and 150 watts through each of the two 8-pin power connectors, but what happens when the TDP is exceeded - I'm now completely in the dark about. BTW - I tried MSI AfterBurner as you recommend, after reading a review about the top 5 GPU over-clocking utilities. The author said that MSI AfterBurner was the gold standard and that EVGA Precision X (which I use uniformly) was 2nd in the ranking. That being said, while the MSI utility may be great for the vast majority of uses and users, it has one critical flaw at which, for this endeavor, make it a bit tarnished. It gave me really crazy numbers for the power draw of one of the GPU processors on the 705 MHz Titan Z that I was testing. As for the second GPU, the numbers were completely in line with what GPU-Z was telling me. However, for the first GPU processor it was giving me numbers in the 700 to 900 % above TDP range. Thus, I think that for judging TDP that I'll stick with GPU-Z, although I do like the other features of the MSI utility better.

I believe that there are some important points that we haven't discussed:
(1) Imagine being in a 40 m deep hole; now imagine it's 80 m deep. Would it normally require your exerting the same amount of energy to climb out of each hole?

(2) The Titan Z card was to leap frog Nvidia so far ahead of AMD in performance as to stop AMD dead in its tracks. Little did Nvidia know what AMD had up it's sleeve until shortly after the Titan Z was announced. Then upon realization that AMD had a very fast GPU answer, even a water-cooled only dual GPU answer, Nvidia held off releasing the Titan Z for about two months while it recovered from the shock and further tweaked the Titan Z and intentionally didn't didn't give samples to a lot of reviewers. The previous dual processor Nvidia cards had not been down clocked as significantly as the Titan Z was eventually down-clocked as to its base speed */. I read some reviews where the reviewer thought that the Titan Z was a 500 watt TDP card (i.e., 2x a Titan Black), but it wasn't and yet its then was cost $2K (US) more. At its standard/base clocking, its a lot less powerful than two Titan Blacks [ In reality, less powerful than two of my original Titans]. I'm sure that there's been some binning madness to find lower base clocked GK110Bs that have a lot of boost potential under the right environment, but the standard/base clock rate is a lot lower than most imagined that it would be. Two GK110Bs fully clocked as Titan Blacks are clocked, would have made the Titan Z (non-Hydro Copper], probably, about a four or five slot card on air cooling - which would have been completely unacceptable. In sum, I believe that the Titan Z can perform like two Titan Blacks ONLY IF one can get around the heat issue and power issues moving from an extremely low base clock. And if one does get around the heat issue by using a really effective water-cooling method one will be far along in the process of reaping the benefits of having two Titan Black processors on the same card, but here's the rub. The chips that excel at that mission ( reaping the benefits of having two Titan Black processors ) probably were thrown only into the Titan Black bin and a Titan Black has a TDP of 250 watts. I'll bet that to get that kind of performance that the higher TDP numbers shown by GPU-Z and MSI (as to CPU2) for blowing past the 375 watt TDP aren't uncommon. So, if you're satisfied simply with performance greater than that of one Titan Black, for a card that has only recently dropped in price from about $2,900 (US) to about $1,500 (US) {and strangely enough, that's what the dual-295 AMD cards used to cost} then you'll likely stay within the 375 watts TDP. But if you want to fully unlock the full potential of those two GK110B's, then you're going to have to break a few eggs by blowing past that TDP of 375 even if you're running Octane Render. To be sure, you will probably not necessarily have to double the TDP of one Titan Black to achieve that goal because all of the chips requiring some power aren't doubled on the Titan Z, but most of the biggest power consumers are - more memory chips and an additional GPU processor and there're a few chips that weren't needed when all that you had was one processor on the card. Also, I recognize that the processors in the Titan Z likely have a lot better boost range than those in the Titan Black, but higher boost range isn't a purely equal substitute for a higher base clock. Thus, I believe that because the Titan Z began in such a deep base clock hole, that I'll have to continue to exceed the stated Titan Z TDP because I want maximum performance from both of my GK110Bs and staying within that 375 watt TDP doesn't appear likely to get me there.

3) I'd bet you that any two of the GPUs in the first line of your signature would out performance any one of my GTX Titan Zs if it was running only on air-cooling, at stock setting and staying within its TDP. Moreover, I wouldn't be shocked if that were the case even when I finish the mod. Sometimes, something can be so below standard that its deficiencies cannot be made up in a particular allotment of time, regardless of the amount of energy expended to make up from it low beginnings.

*/ In fact, even within the 7xx series, only some, but not all, GT series cards have such low base clocks [ http://en.wikipedia.org/wiki/List_of_Nv ... 700_Series ]. Even more significant is that the same comparison applies to the 6xx series[ http://en.wikipedia.org/wiki/List_of_Nv ... 600_Series ], to the 5xx series [ http://en.wikipedia.org/wiki/List_of_Nv ... 500_Series ] and to the 4xx series [ http://en.wikipedia.org/wiki/List_of_Nv ... 400_Series ] ( and I know that the earlier ones didn't have boost capability). So we're talking about an epically low base clock for the Titan Z. How low is it? It's so low that only some (but not all) of the cheapest cards that Nvidia has made in the last 5 years are clocked that low.

[Tutor]

Tutor,

... .

PS. Sorry if I missed it - if there is a limit of 12 gpus in Octane how will you make use of 7 Zs? Will you render on 6 and excessive 1 will handle the system?

If 12 is truly a cap, then I'll like run another instance of Octane on it to handle the excess.

[smicha]

Tutor,

... .

PS. Sorry if I missed it - if there is a limit of 12 gpus in Octane how will you make use of 7 Zs? Will you render on 6 and excessive 1 will handle the system?

If 12 is truly a cap, then I'll like run another instance of Octane on it to handle the excess.

But two separate Octane licences don't give you (over network) 24 gpu limit but still 12, so extra licences have to be used separately. This is a really bad move from otoy.

[Tutor]

Tutor,

... .

PS. Sorry if I missed it - if there is a limit of 12 gpus in Octane how will you make use of 7 Zs? Will you render on 6 and excessive 1 will handle the system?

If 12 is truly a cap, then I'll like run another instance of Octane on it to handle the excess.

But two separate Octane licences don't give you (over network) 24 gpu limit but still 12, so extra licences have to be used separately. This is a really bad move from otoy.

But does that mean that I can't split a render job in two and run two instances of octane on the same machine to render it. I have multiple Octane Licenses. I wouldn't then be using net render. However, I do agree with you 100% that it appears that Otoy's moves get it the way of our having successful businesses. And if that is infact the case I'll move completely to another software provider or just write my own applications.

[smicha]

Slitting the job is fine.