Best Practices For Building A Multiple GPU System

[Tutor]

Tutor,

First of all - hats off, respect!

What fans do you have for the double Aquacomputer radiators? What are their static pressure? Pushing air through such thick fans require about 3mm H2O static pressure fans (or even more). Just asking.

More and more great machines on the forum

I haven't purchased any new fans ones [yet?]. Since my Tyan GPU server currently behaves like a server in every respect, including sounding just like a jet plane because of the Tyan's three thick, large diameter internal turbo fans, I'm thinking about sticking to that theme [but only by as much as necessary to meet my thermal goals] by using my Sunon PMD1212PTB1-A(2).F.GN's that have been pulled from other servers. I am awaiting my fan control box to bring them all under my total control.

Here are some of the specs for my Sunons:
...
Voltage - Rated 12VDC
Size / Dimensions: Square - 120mm L x 120mm H x 25mm W [There'll be nine of them on each MoRa rad., two of them inside Mr. Freeze(r) on the the Alphacool NexXxoS Monstra Dual 120 mm Radiator and four of them inside Mr. Freeze(r) on the 4 ganged together Aquacomputer Airplex Modularity System 240 Radiators]
Air Flow 150.0 CFM (4.24m³/min)
Static Pressure 0.620 inches [or 15.748 mm] H2O (154.4 Pa)
...
Noise 54 dB(A)
Power (Watts) 12.0W
RPM 4500 RPM
Termination 2 Wire Leads
Ingress Protection -
Weight 0.485 lb (219.99g)
Operating Temperature 14 ~ 158°F (-10 ~ 70°C)
Current Rating 1A
Voltage Range 6 ~ 13.8VDC
...

Shouldn't these be powerful enough to help to cool the GPUs and CPUs adequately? Would you recommend more powerful fans? If, so what are they and where can I purchase them?

BTW - Smicha, should I still use Octane 1.0 Beta 2.46B to run the performance test or should I use the latest version of Octane [I'm having trouble finding that old beta]?

[smicha]

Shouldn't these be powerful enough to help to cool the GPUs and CPUs adequately? Would you recommend more powerful fans? If, so what are they and where can I purchase them?

BTW - Smicha, should I still use Octane 1.0 Beta 2.46B to run the performance test or should I use the latest version of Octane [I'm having trouble finding that old beta]?

Tutor, are you kidding me? 4500RPM? Man, 2000RPM fans are more than enough, like noctua industrial, sanyo denki S1212F401 with much lower noise.

Yes, if you want to compare your speed across variuos users use latest version of Octane, i.e., 2.15.

Your machine seems to be a true destroyer - please keep posting. Can't wait to see it in action.

[glimpse]

sorry if You already replied this (but couldn't find an answer) on this thread: How are You going to run more than 8x GPUs? =) Guess not on windows..linux then?

[Seekerfinder]

sorry if You already replied this (but couldn't find an answer) on this thread: How are You going to run more than 8x GPUs? =) Guess not on windows..linux then?

Glimpse,
Remember that 8 GPUs is not an official limit on a single machine. Octane / Windows may well pick up more than 8...
Seeker

[Tutor]

Shouldn't these be powerful enough to help to cool the GPUs and CPUs adequately? Would you recommend more powerful fans? If, so what are they and where can I purchase them?

BTW - Smicha, should I still use Octane 1.0 Beta 2.46B to run the performance test or should I use the latest version of Octane [I'm having trouble finding that old beta]?

Tutor, are you kidding me? 4500RPM? Man, 2000RPM fans are more than enough, like noctua industrial, sanyo denki S1212F401 with much lower noise.

Yes, if you want to compare your speed across variuos users use latest version of Octane, i.e., 2.15.

Your machine seems to be a true destroyer - please keep posting. Can't wait to see it in action.

Thanks for the wise and prompt response. My thoughts were and are that I can use the fan controllers to cut those wind tunnels' RPM by a quarter of the whole, or by a third of the whole, or by one half of the whole, etc. to find the sweet spots for various types of tasks. Since my last post about my config., I decided to add as the eight GPU card to this Tyan build - one of my GTX Titans or 780 6Gs. Thus, I'll have under water 6 Titan Zs, 1 Titan Black and either one original Titan or a 780 6G. Since I use more than one Octane GPU accelerated application, even if one of them is subject to some arbitrary limit on the number of GPUs that it will support, the excess GPUs can be put to use by another GPU accelerated application.

A couple of the Titan Zs that I acquired came as reference design Titan Zs (the 705 MHz version). Before I made any modifications to one of them, I decided to try it out in the Tyan to see how it runs on air. Running Octane Render 2.14, I was able to overclock that Titan Z by 250 MHz and it's ram by 100 MHz ( these werent the limits) 'to achieve in DL 72.94 Ms/sec. av. and in PT 24.58 av. Thoughtout both tests the Titan's temps ran between 52 and 54 degrees (C). Both Coherent Ratio and Path Term were set to 1. They (each of it's GPUs) were boosting on air at over 1350 MHz. I believe that the extended over-clocking capability way due to the Tyan's three internal wind tunnels at play. Just think what boost speeds they should be easily attaining on chilled water, pre-chilled by the equivalent of eighteen single 120x120x65mm radiators represented by the girth, width and height of two Mo-Ra3s ramped down in heat by Sunons!

[Tutor]

sorry if You already replied this (but couldn't find an answer) on this thread: How are You going to run more than 8x GPUs? =) Guess not on windows..linux then?

As I have indicated in my signature, I have "24 tweaked/multiOS rendering systems." That means that they all run the three top OSes, including Linux. However, as you (and Seekerfinder points out immediately after your post), I going to burst through that unofficial, mythological 8 GPU limit one way or the other.

BTW - Here's what my cryptic signature [ oTitan RD TE>83; KE=145K+ CUDA cores; 13K+ ATI Stream PUs; 230 CPU cores. CB11.5-48.5 Win; CB15-3,791 Win; GB2-58K+ Lnx; GB3-71K+ Lnx/49K+ OSX; Sala-12K+. ] further means:
1) The phrase "oTitan RD TE>83" means that together all 24 of my systems have the Octane rendering power of more than 83 original GTX Titans;
2) The phrase "KE=145K+ CUDA cores" means that those 24 systems have the rendering equivalent of over 145,000 Kepler CUDA cores;
3) The phrase "13K+ ATI Stream PUs" means that one of my systems has more than 13,000 ATI stream processing units;
4) The phrase "230 CPU cores" means what it says in regards to the total CPU rendering capability of my network;
5) The phrases "CB11.5-48.5 Win; CB15-3,791 Win" means that my two identically configured 32-core Supermicros score over 48.5 CPU points in Cinebench 11 and over 3,791 CPU points in Cinebench 15 [ http://www.cbscores.com ];
6) The phrases " GB2-58K+ Lnx; GB3-71K+ Lnx/49K+ OSX;" means that those same Supermicros score more than 58,000 points in Geekbench 2 when running Linux [ http://browser.primatelabs.com/geekbench2/top ] and 71,000 points in Geekbench 3 when running Linux [ http://browser.primatelabs.com/geekbench3/multicore ] and even when MacOSX is forced to run on them, they are not to be pitied for they score over 49,000 points [ http://browser.primatelabs.com/geekbenc ... core_score ];
7) The phrases "Sala-12K+" [ http://www.luxrender.net/luxmark/ ] refers to my other Tyan server now, but it used to refer to the OpenCL rendering prowess of this one before the Titan Z's evicted the regular/original Titans to my other Tyan server and that other Tyan server's GPUs were accordingly pushed down to some of my other systems.
8) What they all mean is that I mold my systems to be the best that I can make them be and that I'm not enslaved to any particular OS or system manufacturer or software provider. Of those 24 systems, my 4 MacPros, are the ones that I have had the least involvement in shaping; but I did shape them [ see, e.g., http://www.computerworld.com/article/24 ... -pro-.html ] . My 2 Supermicros and 2 Tyans were purchased as barebones systems, e.g., without memory, storage, CPUs and GPUs. Moreover, as to my other (and mostly newer) 16 systems that I have built from the ground up, each of them is the fastest Geekbench tested system of its exact configuration [see, e.g., [ http://browser.primatelabs.com/user/9077 ] and http://www.insanelymac.com/forum/topic/ ... ch-scores/ ].

Many thousands of years ago our ancestors were separated and we changed and were changed to better cope with our then new environments, but we are all one people at bottom-line, and are all cousins. Now is dawning a time when I can easily renew my familial relationship with my cousins (in alphabetical order) Bulwerk, Glimpse, P3taoctane, Rappet, Seekerfinder, and Smicha, as well as all of the rest of you, for, to me, that is the most enduring, significant and special part of this life's journey. As my cousins feed me, I feed them and others cousins; then other feed me and I feed others, then ... . [ http://forums.macrumors.com/showthread.php?t=1333421 ]. Let's us not ever forget that we are one. This is how this old man rolls [ http://forums.macrumors.com/member.php?u=332616 ] and will continue to do so for as long as his creator allows.

[smicha]

Tutor,

I'll be honest with you - as you started your posts they seemed too long to me, filled with so many words. I thought to myself - why the guy cannot shortly express himself. But now I see how foolish I was. This post changed something in me and I realized that there is someone not only with supercomputers out there, but with great wisdom. I read the 1-8 points twice, carefully. With every sentence my eyes were opening more and more. The computational power is like a dream for me, dream that will never come true. But it's good to know you are my cousin

If I may ask - are these all servers for 3D work, how many people make use of them?

[Seekerfinder]

2) The phrase "KE=145K+ CUDA cores" means that those 24 systems have the rendering equivalent of over 145,000 Kepler CUDA cores;

There are render farms and, apparently, there render farmers... Hats off to Tutor!

Seeker

PS: when the apocalypse comes, I shall seek out my cousin, Tutor. With his computer power we could probably restart the world. No? Ok, I'll settle for the matrix...

[rappet]

I cannot quote tutor, because then every post will be one page of this thread...
Therefor I just want to says this on the words of tutor: it is pure poetica !

Edited: @tutor,
I am not a workplace problem solver, management trainer, systems analyst, network administrator, software developer, inventor, universal repairman, plumber, electrician, gardener, nutritionist, bodybuilder, b-ball player, musician, artist or chef, but I can say I am an ex-bartender as well.
Please do add poet to your occupation profile, because you are.

[Tutor]

Tutor,

I'll be honest with you - as you started your posts they seemed too long to me, filled with so many words. I thought to myself - why the guy cannot shortly express himself. But now I see how foolish I was. This post changed something in me and I realized that there is someone not only with supercomputers out there, but with great wisdom. I read the 1-8 points twice, carefully. With every sentence my eyes were opening more and more. The computational power is like a dream for me, dream that will never come true. But it's good to know you are my cousin

If I may ask - are these all servers for 3D work, how many people make use of them?

Thanks Smicha - your are an excellent tutor to me. I'm my only employee. Only I use of those systems. I use them mainly for 3d animation, music production, video editing, compositing, programming and game creation. I usually control them through my MacBookPro.

One reason why my posts are so long is because I try to err on the side of giving as much detail as I can. To me, that allows others, even neophytes, to replicate exactly what I've done and know why I chose to do it that way. So, I can understand your feeling that my posts are too long. That's why you are my tutor.

If you ever need my help, all that you have to do is let me know and I'll give to you my very best efforts.