Best Practices For Building A Multiple GPU System

[Tutor]

2) The phrase "KE=145K+ CUDA cores" means that those 24 systems have the rendering equivalent of over 145,000 Kepler CUDA cores;

There are render farms and, apparently, there render farmers... Hats off to Tutor!

Seeker

PS: when the apocalypse comes, I shall seek out my cousin, Tutor. With his computer power we could probably restart the world. No? Ok, I'll settle for the matrix...

Thanks Seeker,

We're already seeking each other out and I hope that we continue. Our realization that we are all one can restart the world even before the apocalypse, and our acting towards each other, in everything that we do, as a vital part of one people could render the apocalypse a nullity.

[Tutor]

Act it, Live it - That's the definition of truth.

I cannot quote tutor, because then every post will be one page of this thread...
Therefor I just want to says this on the words of tutor: it is pure poetica !

Edited: @tutor,
I am not a workplace problem solver, management trainer, systems analyst, network administrator, software developer, inventor, universal repairman, plumber, electrician, gardener, nutritionist, bodybuilder, b-ball player, musician, artist or chef, but I can say I am an ex-bartender as well.
Please do add poet to your occupation profile, because you are.

Thanks Rappet,

I'm sure that there are many things at which you excel - I can sense them in your words, with modesty and kindness being just a few. May you let all of your lights shine brightly worldwide. Some may think that technology and personal character do not mix. Sometimes, however, they mix in ways with boundless repercussions on billions of our cousins. Rappet, let your talents continue to mix for their good.

As for me, I worked as a bartender while I attended college at Columbia University in New York City in the early 1970s. My mother began teaching me cooking and gardening when I was age 5. I began studying art and music when I was age 6. I began weight lifting and studying nutrition when I was age 11. I began playing basketball when I was age 13. In high school in Los Angeles I ran track (100 yard dash and 400 yard relay) and field (shot-put and high jump). In college, I played football. All my other activities (i.e., workplace problem solver, management trainer, systems analyst, network administrator, software developer, inventor, universal repairman, plumber, electrician) began after I reached adulthood. At age 61, I still play basketball and pump iron regularly, as well as do my own home repairs, plumbing and electrical work. I have the largest garden in my neighborhood and share its production with my larger family, including my neighbors and others. I continue to work as a workplace problem solver and management trainer for other businesses, and am now my own systems analyst, network administrator and software developer. Also, after I was told that I had diabetes, I developed a drink that led to my physician taking me completely off of insulin and all other related medications. Now I can eat whatever I want to eat. I expect to begin marketing my drink as soon as I phase out of my working for others. In essence, I'm working on devoting the next phase of my life exclusively with feeding and improving my mind, body and soul and allowing the creative parts of me to flourish. I will never stop being Tutor - the nickname given to me by my junior high school friends when I began tutoring younger children in mathematics. I tutored children while I was in high school, while I was in college and while I was in graduate school and still currently tutor children - now in electronics and technology.

[Tutor]

then, he inspired me to feed it correctly.

Smicha,

Following up on one of my posts and one of yours, you made me think more critically and learn a lot more about the power requirements of all of my systems. I know that you, like I, try to safely extract all of the power from a system that it has available to offer. To do that includes making enough electrical power available to all of the system's components. Here's how your first post relates to my last update where I installed the RD Titan Z for testing on air. When I was over-clocking that GPU and testing it using Octane's Benchmark scene, GPU-Z was telling me that the Z was regularly requiring over 120% of it's TDP. The TDP for a RD Titan Z is 375 watts, indicating to me that to make that Z perform as it was then performing was requiring about 450 watts to run that OctaneRender benchmark as the GPU was then clocked. I've read reviews of the Z indicating that, for certain tasks, it was consuming more than 500 watts, even as high as 575 watts. Thus, based on your prodding me to think more deeply about adequate power given what I'd be doing, I'm configuring this Tyan mod to allow, at least, up to 600 watts to each Z, 400 watts to the Titan Black and 400 watts to the the GTX 780 6G (or RD Titan, if I finally chose to use it). [Luckily, the EK water block that I've ordered for my eighth GPU fits both the Titan and the GTX 780s.] The 600 watts to be allocated to each of the Titan Zs and the 400 watts to be allocated to the Titan Black and to the eighth GPU represents 1.6 times their stated TDP. Maybe I getting a little bit too optimistic, but my instincts tell me that if I can over-clock a factory Titan Z set to run each GPU processor with a base speed of 705 MHz, to run with an over-clocked base speed of 955 MHz on air, that it is likely that I can get even higher stable over-clocks by water-cooling the GPUs, if I have sufficient power available to go along with that cooling and I have capable GPU processors. Having a lot more GPU overclocking experience than I, I truly value your experience, insights and prognostications. So, what would you expect to be the type of performance delta there is of water-cooling vs. air cooling, plus especially if the water-cooling results in sub-ambient air GPU processor temperatures, particularly under load. Are my wattage allocations much too high?

BTW : 955 MHz / 705 Mhz = 1.35. Moreover, it's been my experience with CPU over-clocking that there's a point at which eking out the max, safe over-clock requires a lot more power than it did for lesser initial over-clocks.

[Seekerfinder]

The 600 watts to be allocated to each of the Titan Zs and the 400 watts to be allocated to the Titan Black and to the eighth GPU represents 1.6 times their stated TDP.

Tutor,
If I can chime in here, you've probably picked up on these forums that Octane typically uses less power than most games or other intensive applications - it seems only about 70-75% (an unscientific guestimate). My theory for this is that the v-ram is less active once the scene is loaded in Octane.

So if you plan to use those GPU's for other, more intensive tasks as well as octane, by all means your suggested power route seems to be the safe option. But, even with your intended overclock margin, I would think that providing the stated manufacturer TDP should be adequate for Octane use.

I'd also like to hear Smicha, Glimpse (& others') opinions on this.

BTW, how do you get past the measly 110V typical in US? Or do you have 220V or - probably with your setup! - 3 phase power? I would not want to pay your company's power bill...

Best,
Seeker

[Tutor]

The 600 watts to be allocated to each of the Titan Zs and the 400 watts to be allocated to the Titan Black and to the eighth GPU represents 1.6 times their stated TDP.

Tutor,
If I can chime in here, you've probably picked up on these forums that Octane typically uses less power than most games or other intensive applications - it seems only about 70-75% (an unscientific guestimate). My theory for this is that the v-ram is less active once the scene is loaded in Octane.

So if you plan to use those GPU's for other, more intensive tasks as well as octane, by all means your suggested power route seems to be the safe option. But, even with your intended overclock margin, I would think that providing the stated manufacturer TDP should be adequate for Octane use.

I'd also like to hear Smicha, Glimpse (& others') opinions on this.

BTW, how do you get past the measly 110V typical in US? Or do you have 220V or - probably with your setup! - 3 phase power? I would not want to pay your company's power bill...

Best,
Seeker

Thanks Seeker,

I make do with 110V. It wasn't until I experienced an 8K modem that a 4K modem felt really slow. Not having 220V is probably a bit like that.

GPU-Z may be giving me incorrect information about the power being consumed, but it's been and is telling me that when I significantly over-clock a GPU in Octane, FurryBall or Thea Render [and CPU-Z does the same thing when I do the same for a CPU, but of course in other CPU related applications] that it's drawing higher power for the GPU [CPU] by X% above the TDP. When I do not over-clock the GPU, GPU-Z almost always shows power consumed below the TDP for those three applications. In fact, to significantly over-clock a CPU or GPU some upward adjustment of allocated voltage is usually required to prevent over-clocking related failure, i.e, Octane's crashing, the GPU's stepping down it's speed significantly, or Nvidia driver error occurs - all requiring a system re-boot. On the other hand, I do believe that Octane's power load is less than some other intensive GPU activities and GPU-Z confirms that to be the case.

[smicha]

Tutor,

Here http://render.otoy.com/forum/viewtopic. ... 0&start=80
I posted power draw results for 780 6GB and for Titan - stock and OC (all watercooled). From my measurements it occurs that Titan draws about 170-225W overclocked in Octane; 780 6GB - 150-180W in Octane, but fully stressed in Furmark it draws 300W.

Based on these measurements I came to conclusion that the simplified general rule for PSU nominal power may be approximated by: (#of GPUs+1)*300W, although Octane will draw (#of GPUs+1)*200W. This should give some reserves for spikes (as my power draw meter shows peaks at 1500W).

As for GPU-Z: please install MSI Afterburner and monitor power usage there. For 780 I scored 68-77%, for Titan 85% (or more).

Your Titan Z shall go to 1250mHz on core at stock volts (1.2V I believe) easily (watercooled) - please see Bulwerk build here: http://render.otoy.com/forum/viewtopic. ... 2&start=20
(he reported 1250mhz OC and 99% power usage).

I also estimated an OLS model (log-log scale for elasticity) that showed 3/4 and 1/4 of overclocking power coming from core and memory respectively
http://render.otoy.com/forum/viewtopic. ... 0&start=70

If you have any more questions all answer them with pleasure.

PS. I'll PM you as for diabetes.

[Tutor]

Tutor,

Here http://render.otoy.com/forum/viewtopic. ... 0&start=80
I posted power draw results for 780 6GB and for Titan - stock and OC (all watercooled). From my measurements it occurs that Titan draws about 170-225W overclocked in Octane; 780 6GB - 150-180W in Octane, but fully stressed in Furmark it draws 300W.

Based on these measurements I came to conclusion that the simplified general rule for PSU nominal power may be approximated by: (#of GPUs+1)*300W, although Octane will draw (#of GPUs+1)*200W. This should give some reserves for spikes (as my power draw meter shows peaks at 1500W).

As for GPU-Z: please install MSI Afterburner and monitor power usage there. For 780 I scored 68-77%, for Titan 85% (or more).
... ..

Smicha,

Just for me to be clear in my understanding of your formula for Octane use, the 300/200 watts figures include the GPU card's total power draw - and that includes the draw from the PCIe slot in which the card sits and from both of the PCIe power sockets on top of the card into which PCIe cables are connected. Is that correct?

Thanks, Tutor

[glimpse]

Hi, Tutor.

I knew You were crazy (in a best possible way =)

anyway, happy to help with anything & happy to be in the same boat (forum),
as we all learn from each other here - more discussion the better is outcome!

Looking forward for Your build! & fingers crosse Your idea would work-out!
Insane build & there'll be mind-blowing numbers!

tom

[smicha]

Tutor,

I have no knowledge how the power draw is distributed - whether this is via a pcie slot or power connectors. The (#gpus+1)*300W rule allows me for rough approximation of PSU nominal power I shall get to run certain number of gpus safely. For example, for 7 Titans Z I would go with 3x1500W PSUs (15x300W), although entire system while rendering in Octane may draw about 3000-3300W (14x200W for gpus + 200-500W for the rest). So with 3x1500W PSUs there should be plenty of safety/stability reserves (even for heavy overclocking).

PS. Sorry if I missed it - if there is a limit of 12 gpus in Octane how will you make use of 7 Zs? Will you render on 6 and excessive 1 will handle the system?

[Seekerfinder]

I have no knowledge how the power draw is distributed - whether this is via a pcie slot or power connectors.

The mobo supplies 75w through each PCIe slot. The rest is auxiliary power straight from the PSU.

Seeker