Best Practices For Building A Multiple GPU System

[smicha]

What is the price for this beast?

[FrankPooleFloating]

Okay Tutor... now I have to ask: What in gawds name are you doing with all these beasts?.. Farming?.. Mining?.. Folding?....... Or maybe just a filthy rich bloke with a crazy-expensive hobby? Please do tell. Leave nothing out.

[rappet]

Okay Tutor... now I have to ask: What in gawds name are you doing with all these beasts?.. Farming?.. Mining?.. Folding?....... Or maybe just a filthy rich bloke with a crazy-expensive hobby? Please do tell. Leave nothing out.

Yeah... I understand where Frank's question is coming from... I am curious as a cat too

[Tutor]

What is the price for this beast?

I can dream, can't I?

This beast is beastly priced (all in USD) in a minimum config. w/4 GTX Titan Xs for $24,750 and goes up to ~ $39,800 for 16 Titan Xs. I was told that if they were to sell just the chassis without any GPUs (which they won't do), it would cost about $19,000. They sell the air-cooled RD Titan Xs for about $1,200 each. The chassis requires special power connection cables and supports (included in the price for just the no. of GPUs that one pays to be pre-installed by them).

Now that I've been fully awakened :

On a related, but a lot more generally applicable, note, if you have read (or ever feel like reading) the Infosec Institute articles that I referenced on the last page [ http://render.otoy.com/forum/viewtopic. ... 87#p241271 ], pay special attention to discussions of "BARs." Why? The One Stop Shop representative stated that which I didn't won't to hear today after just digesting the price quote, but have been suspecting that it is an essential ingredient in maximizing the number of GPUs that a system is capable of handling, namely that Windows (Server, XP , Pro, 7-10, etc., i.e., all) essentially sucks wind when it comes to allowing nine or more GPUs to operate successfully in a multi-GPU single system setting because it will not allow easy BAR adjustments. "There is a BAR1 Register limitation that is fixed in {W}indows and will prevent you from going back 8 GPUs per server. Using Linux would be required to get beyond 8 GPUs." That's not to say that such bar adjustments are impossible, but they are just truly harder to do. **/ BAR adjustments hold the key to successfully managing/adjusting IO space*/. LINUX IS KING of this realm [ and note that ZZZ1000 used Linux to run 18 GPU processors via CUDA 5.5 [ https://devtalk.nvidia.com/default/topi ...s/#4030904 ]. ]. I'm glad that almost all of my systems run the major OSes, Windows and Linux included, but the fact that C4d doesn't run natively on Linux troubles me.


*/ " PCI device registers (outside of the PCI configuration register) and memory (RAM and/or ROM) can be located anywhere within the CPU memory address space (or CPU I/O address space) because PCI devices contain programmable registers, called a Base Address Register (BAR). Figure 1 shows the location of the BARs in the header portion of the PCI configuration registers. A BAR controls where the PCI device registers (outside of the PCI configuration register) and where memory is mapped in the CPU memory, or I/O address space. The reason for this programmability is to eliminate address conflicts between devices, which plagues the old ISA and EISA devices." [ http://resources.infosecinstitute.com/p ... nsion-rom/ ]

**/ Per Gordan79 on EVGA's forum in discussing getting 8 or more GPUs running on the EVGA-SR2 [ http://forums.evga.com/Evga-SR2-with-8- ... px#2328686 ]:
"The only setting in the BIOS that is relevant to running many GPUs is the memory hole (which is settable to a maximum of 3GB [Note well - This limit applies to the motherboard that we were discussing at that time, namely, the EVGA SR-2]). No other setting is relevant to running many devices with large I/O memory requirements. There is no way to influence where the BIOS maps the different I/O memory blocks, so it will either work or it won't, and if it doesn't, your only hope is to start modifying the soft straps on the VBIOS.

Those are documented here:
https://indefero.0x04.net/p/envytools/s ... straps.txt (copy is attached below)
The bits you would need to adjust are:
BAR 0: bits 17,18,19
BAR 1: bits 14,15,20,21,22
BAR 2: bit 23

To adjust them, dump the BIOS and look at the location appropriate to your GPU to get the current strap (remember to swap the byte order) and modify it accordingly. Flash the new strap to the GPU with nvdlash --straps. Getting it wrong will in many cases result in a bricked GPU and you will need to unbrick it by booting it with the BIOS chip disabled, then re-flash it. If you are planning to experiment with this I highly recommend soldering a switch and a resistor (in series) across the VCC and GND for easy unblocking." Also, note well that Gordan is describing a GPU by GPU bios modification. That's why I say it's harder to do.

Okay Tutor... now I have to ask: What in gawds name are you doing with all these beasts?.. Farming?.. Mining?.. Folding?....... Or maybe just a filthy rich bloke with a crazy-expensive hobby? Please do tell. Leave nothing out.

I declare that I'm a filthy dressing, 3d render farming bloke using all of my beasts for mining 3d projects payable with foldable currency that I use to support pastimes/hobbies, such as helping the impoverished and others, though I could do more if I were rich.

P.S. I'm known as Mr. Cheapo when it comes to purchasing things. I hope that I didn't leave anything out.

[Notiusweb]

Just a side step for a second:

Tutor, I think your rig's power plan looks to have a good, and conservative, distribution method. To anyone out there, 12 gauge wire is thicker than 14 gauge (or higher), and thicker is used for more heavy duty electrical appliances. Most bitcoin forums recommend 12 or 14 gauge. I found that on Octanebench, my entire rig trips a single 15A breaker when overclocked, but does not if run at stock. So, one could mount a pretty good idea with how viable a power set up is by trying to max out their cards during Octane Bench. But, if divided up,I can overclock the complete rig with 2 of my Z's powered on a separate breaker outlet, via a 50ft 14 gauge 15A extension cord. I am able to do this because I have 2 separate external PSUs, in addition to the PC's own PSU, to work with.

by Tutor » Mon Jul 13, 2015 10:04 pm

**/ BAR adjustments hold the key to successfully managing/adjusting IO space*/.

Hmmmm...Very interesting!.....Thanks!

[Tutor]

Just a side step for a second:

Tutor, I think your rig's power plan looks to have a good, and conservative, distribution method. To anyone out there, 12 gauge wire is thicker than 14 gauge (or higher), and thicker is used for more heavy duty electrical appliances. Most bitcoin forums recommend 12 or 14 gauge. I found that on Octanebench, my entire rig trips a single 15A breaker when overclocked, but does not if run at stock. So, one could mount a pretty good idea with how viable a power set up is by trying to max out their cards during Octane Bench. But, if divided up,I can overclock the complete rig with 2 of my Z's powered on a separate breaker outlet, via a 50ft 14 gauge 15A extension cord. I am able to do this because I have 2 separate external PSUs, in addition to the PC's own PSU, to work with.

So true: how one divides the power consumption affects the ability to deliver power where it's needed most and for those like us who like to tweak our GPUs proper power allocation is imperative.

by Tutor » Mon Jul 13, 2015 10:04 pm

**/ BAR adjustments hold the key to successfully managing/adjusting IO space*/.

Hmmmm...Very interesting!.....Thanks!

To be absolutely clear, BAR adjustments (that will help to allow one to maximize the number of successful GPU processor installs) can be made at the system level through system bios for all of the GPUs or, if the bios doesn't allow I/O space adjustment [usually through activating a 4G option], then one can adjust the BARs directly at the individual GPU level [but that is a PITA]. Moreover, until Windows is improved for the task, when the GPU count reaches about nine or ten in Windows (version no. matters), to go higher one may need to switch to Linux to fully max out the GPU count. [ http://render.otoy.com/forum/viewtopic. ... 50#p242804 and http://render.otoy.com/forum/viewtopic. ... 50#p238278 ] Also, keep in mind that Octane, unlike some other 3d rendering environments {see for e.g., http://furryball.aaa-studio.eu/support/faq.html }, has a 12 GPU limit per license.

If your GPU is disappearing and reappearing, you might try running this test: https://msdn.microsoft.com/en-us/librar ... s.85).aspx .

[Notiusweb]

Post by Tutor » Mon Jul 13, 2015 10:04 pm

I can dream, can't I?

Once you add the GPUs to the rig and get them working, the dream will really come to life when you learn the biology of the system such that you can maximize its potential.

Also, keep in mind that Octane, unlike some other 3d rendering environments..., has a 12 GPU limit per license.

Tutor, I hit 962 avg score on Octane Bench with max of 989....can't wait for your monster rig to leave that in the dust! I'm guessing 1,200 - 1,300...or 1,500?
Once you do that "in-house", it forwards the case that a 12 GPU limit is an outmoded business model, especially if the model claims to be cutting-edge.

[Tutor]

An imaginative mind is a terrible, terrifying and dangerous thing to waste, especially when it comes to sandwiches - mozzarella?

Post by Tutor » Mon Jul 13, 2015 10:04 pm

I can dream, can't I?

Once you add the GPUs to the rig and get them working, the dream will really come to life when you learn the biology of the system such that you can maximize its potential.

Truer words were have never been spoken.

Also, keep in mind that Octane, unlike some other 3d rendering environments..., has a 12 GPU limit per license.

Tutor, I hit 962 avg score on Octane Bench with max of 989....can't wait for your monster rig to leave that in the dust! I'm guessing 1,200 - 1,300...or 1,500?
Once you do that "in-house", it forwards the case that a 12 GPU limit is an outmoded business model, especially if the model claims to be cutting-edge.

Notiusweb,
Congratulations! I thought that those highest ones were your system's scores. I'm looking forward to your growing them even higher by getting greater cooling and/or more power headroom for your tweaks. I can see that you are hungry and always have quite an appetite. I'm going to let you, and only you, in on secret dish, so don't let anyone else in on it. It's coded below, in-line just before any GPUs -

(I) The average score for one Titan X is ~ 129 points. The average score for one Titan Z (both processors) is ~ 173 points. Thus, you've bested the average scores for individual GPUs of the same kind.

(1) 173/2 = 86.5 [1/2 Z]
(2) 4x173 = 692 [4 Zs]
+(3) 129 [1 X]
907.5

(II) The peak single GPU scores for those GPUs are:

1) 185/2 = 92.5 [1/2 Z]
(2) 4x185 = 740 [4 Zs]
+(3) 150 [1 X]
982.5

Thus, you've bested the high scores for individual GPUs of the same kind. Again, congratulations! Thats quite a feat.

Now for the most discrete of tests [I like numbers so big that accuracy demands use of a comma {or two}]:

A GTX Titan Z has exactly 2x number of CUDA cores as a Titan Black. To the extent that you can replicate the performance of two Titan Blacks in one Titan Z your scores can be even higher. To be sure, one Titan Black GPU processor has a relatively higher base clock than either of the two GPU processors of the Titan Z {889 [Black] vs. 705 [Z]} yielding a higher boost and higher filtrate rates, along with proportionately higher Gflops for the Black on a single processor basis {5,121 vs. 8,122}. Higher clocks, to a significant degree, are captives of power and cooling. The Titan Black has a relatively higher TDP {250W (Black) v. 475W (Z) {2x250=500}. The Black is very likely running cooler than the Z {being the only GPU processor on the card}. So increasing the cooling and inching up the power to your Zs might yield scores closer towards the following:

High score for one Titan Black = 116.43; your system has the equivalent of 9 of them. Thus, I'd set the following goal to getting closer to 1,200 OctaneBench Pts.:
(1) 116.43 x 9 = 1,047.87 for 9 GPU processors of Zs
(2) + 150 (X)
1,197.87.

Can you increase the GPU spacing and GPU/environmental cooling? That way, there would be no taste of dust, unless the wind is blowing against your backside. Know what - all of these calculations have made my temperature rise and given me a voracious appetite. I think that I'll make a cool sandwich (or two or three). Also, I like more bread than mozzarella cheese. I like my cheese thin and cut into a perfect square and sandwiched tightly (as if glued ) between two larger, cooling, and screwed together slices of bread (one wet and the other airy).

Resources:
(1) https://en.wikipedia.org/wiki/List_of_N ... 700_Series;
(2) http://www.amazon.com/Arctic-Silver-Alu ... dhesive+5g ;
(3) http://www.amazon.com/s/ref=nb_sb_noss? ... r+12V+5.8A ;
(4) http://www.ebay.com/itm/4-Circuit-Wire- ... 339f6dca16
(5) http://www.amazon.com/Universal-Compute ... ner+Channe ; and
(6) http://www.directron.com/r24.html [ This one fits the sandwich if you use no. 5, but you'll also need 4 screws, 4 washers and 4 nuts to keep the sandwich safely intact.]
(7) https://www.youtube.com/watch?v=Ipt8xqKbCSw & https://www.youtube.com/watch?annotatio ... nKlIc6oCCg & https://www.youtube.com/results?search_ ... cooler+GPU .

By all means keep this observation a secret too: More than one mozzarella sandwich (each in-line) may be most satisfying.

[FrankPooleFloating]

(the following is a compliment) Notiusweb, if that is you in your avatar, you really have an Ian Curtis/David Byrne thing going there. Two of my very favorite singers! Start up a post-punk cover band and come play Akron please.

[Notiusweb]

Postby FrankPooleFloating » Thu Jul 16, 2015 6:11 pm
(the following is a compliment) Notiusweb, if that is you in your avatar, you really have an Ian Curtis/David Byrne thing going there. Two of my very favorite singers! Start up a post-punk cover band and come play Akron please.

LOL...that's me trying to imitate the look that Poser/Daz Studio 3D models have in render test pics in various user forums. It's always a semi-blank stare off to the side. Grew up hearing a lot of Talking Heads, later Joy Division.
But don't get me started on music production, I'll go on all day. Yep, I'll start a punk band called "12th Core", after a certain program's GPU limit .