Best Practices For Building A Multiple GPU System

[smicha]

That's sick!

What exactly have you changed to the bios? Do you mind sharing a photo of the machine?

PS. Is this necessary to keep a gpu in the first slot to make it work (not on a riser)?

[Rikk The Gaijin]

Once I got 11th GPU working I added another Z as I have been having success with these, giving me the 12th and 13th GPU cores

That's INSANE!
You should definitely upload the result on the Octane Bench page...

[Notiusweb]

Post by smicha » Tue Sep 01, 2015 7:56 am

What exactly have you changed to the bios? Do you mind sharing a photo of the machine?

PS. Is this necessary to keep a gpu in the first slot to make it work (not on a riser)?

(1) I disabled LSI storage controller in Southbridge (a secondary SATA interface), this looks to have freed up the PCIE lanes I needed. Running storage only off the motherboard's exclusive SATA ports.
(2) Not pretty, but here it is (my earlier pics in other forum titled 'External Graphics Cards PC' had showed my rig with only USB 3.0 risers (4 Z). Now I have those (3 Z) and the Amfeltec (3 Z). Both are inherently not optimal, as are 1x (vs a Mobo connection of 4x, 8x, 16x, etc.) itou31 has recently noted his findings comparing PCIE connection speed in 'External Graphics Cards PC'.

Photo Sep 01, 9 43 21 AM.jpg

Photo Sep 01, 9 44 09 AM.jpg

Photo Sep 01, 9 46 55 AM.jpg

(3) In my case using the USB 3.0 risers the Titan X would not run as main display off the PCIE riser. Mobo error would come up. However, others (like users Polish Ginger, itou31, and Kandori07) have stated that they were able to run a primary display off a riser.


Thanks to Tutor immensely for this post. I don't know how many times I read this post, and his (Tutor's) posts on 'External Graphics Cards PC' to give me guidance in working my rig.

[Tutor]

Hello!

Continued messing with BIOS and have been able to successfully boot into Win 7 64 Bit with 13 GPU cores on the ASRock X79 Extreme 11.
Main Display Titan X direct on Mobo, 6 Titan Z on risers (3 the USB 3.0 PCIE risers/3 on the Amfeltec 4-way GPU Cluster).

NVid Ctrl Panel5.jpg

OctaneRender6.jpg

Photo Aug 31, 1 53 05 AM.jpg

Once I got 11th GPU working I added another Z as I have been having success with these, giving me the 12th and 13th GPU cores (I considered 2 more Titan Xs, but felt I was capped at 6GB anyway...).
I attempted to post scores on Octane Bench however for some reason they are not uploading (I started a separate post to see if anyone could share thoughts on that.) It does let you select the full 13 GPU on Octane Bench, but it seems the scores match as if it were only 12. As such I am inclined to believe that it is only 12 that are being run in the benchmark even when you select 13.
I also put in a request to see if MSI Afterburner could allow up to and including 13 GPUs via its UI.

I have the Creative Soundblaster X-Fi soundcard still running for music production on my slot Lane 7, and both USB 2.0 and 3.0 ports are active. I have been using my old 660 Ti as the testing card for all this, contemplating that further experimenting may get 14 GPU. To summarize, I have a 1500 W for the Mobo, then 2 separate 1600 W EVGA external power supplies. One is extended to a separate electrical circuit via a 14 gauge extension chord.

CONGRATULATIONS - JOB WELL DONE! Another myth gets exploded about limits of Windows to support many GPU processors. Please tell us what bios tweaks were the key and how you made them?

I'm penning this reply on my Tyan Server (eight x16 slots + one x4 slot + two x1 slots) - my starter massively expansive GPU rendering system. I have not yet installed Nvidia drivers, CUDA or Octane Bench in the Tyan as reconfigured to fully exploit Linux; however, it boots with 7 GTX 590s (each, of course, dual GPU processors) and one GTX 480 [all EVGAs since I have an addiction problem - I'm an EVGA addict]. While I have no illusion that it'll be achieving any scores in your neighborhood, it should be a good Thea and RedShift3d renderer since these applications allow me to render with as many GPU processors as I can load successfully in one system. Moreover, if I can get the rendering equivalent of about 800 OctaneBench points with all used (and old) GPU cards that I got for under $2.5K total, it'll make me a happy puppy. I'll keep you posted about whether and how 15 GPU processors work and perform.

Imagining the glory of 12 Titan Xs...then imagining 12 'Pascals'...

What! Only twelve??????? I hope to get close to 20 GPU processors working in each of my four Supermicros.

[smicha]

It's high time to convince Otoy to increase the 12 gpu limit to...? 16? 20?

[Tutor]

It's high time to convince Otoy to increase the 12 gpu limit to...? 16? 20?

I agree, but how about a limit of 25? Currently, only 12 GPU processors can participate in the OctaneBench render, no matter how many one selects [compare pics and attachment]. Moreover, while they're at it they should move that logo at the bottom lefthand side of the OctaneBench screen because it's blocking the ability to select 16 or more GPUs to participate in the OctaneBench render. That's right - under Linux that old (Circa: 2009-2010) Tyan Server that helped to spur me to create this thread, can now support at least 16 GPUs - Eight GTX 590s. So even though I couldn't perform the test that I had wanted to, the Tyan Server + Linux has allowed me to have a Thea rendering system that has the GPU rendering capability of over eight GTX 980s (630/6=105; 105x8=840).

[Notiusweb]

by Tutor » Tue Sep 01, 2015 7:15 pm
Please tell us what bios tweaks were the key...

Hello Tutor! Yes sir.

First, to anyone, read every post ever posted by Tutor! You'll get an amazing level of evidence-based insight and knowledge.

BIOS
(1) Northbridge - enabled Above 4G Decoding (per your recommendation in an earlier post)
(2) Southbridge - disabled (a) Marvell ESata, and (b) LSI Controller

Sounds simple right? Well, in hindsight it is of course, but I had storage linked to LSI, making it seem like I needed it enabled. Once I reviewed what the piece actually was (an additional set of high speed SATA ports), I learned I could move drives connected to LSI to the traditional Motherboard Sata, and then disable LSI. This freed up PCIE lanes. Now, I can always re-enable, just will cost me a Z or two. But as of now I have the whole system intact with all GPUs.

and how you made them?

Tutor, I know you are asking about a parameter either being enabled, disabled, etc. But, to answer the question philosophically, and to quote an old commercial on using marijuana, "I learned it by watching you"...my rig has been hugely impacted by to the wealth of information you provide everyone.
A HUGE ongoing "Thank You" to you, seriously .

[Notiusweb]

Postby smicha » Tue Sep 01, 2015 9:09 pm
It's high time to convince Otoy to increase the 12 gpu limit to...? 16? 20?

Smicha, do you think there is a methodology behind the limit?
I mean okay, it would have been easy to say, "Windows only allows 7-8 GPU anyway," but putting that type of idea aside now, is there an inability, do you think, on the part of the source code to support larger than 12?
Or would it likely be a business decision?

And also, what is the deal (Jerry Seinfeld voice) with the Octane Render 3 demo ?!
But seriously, what is that GRID K520?

[Tutor]

by Tutor » Tue Sep 01, 2015 7:15 pm
Please tell us what bios tweaks were the key...

Hello Tutor! Yes sir.

First, to anyone, read every post ever posted by Tutor! You'll get an amazing level of evidence-based insight and knowledge.

BIOS
(1) Northbridge - enabled Above 4G Decoding (per your recommendation in an earlier post)
(2) Southbridge - disabled (a) Marvell ESata, and (b) LSI Controller

Sounds simple right? Well, in hindsight it is of course, but I had storage linked to LSI, making it seem like I needed it enabled. Once I reviewed what the piece actually was (an additional set of high speed SATA ports), I learned I could move drives connected to LSI to the traditional Motherboard Sata, and then disable LSI. This freed up PCIE lanes. Now, I can always re-enable, just will cost me a Z or two. But as of now I have the whole system intact with all GPUs.

and how you made them?

Tutor, I know you are asking about a parameter either being enabled, disabled, etc. But, to answer the question philosophically, and to quote an old commercial on using marijuana, "I learned it by watching you"...my rig has been hugely impacted by to the wealth of information you provide everyone.
A HUGE ongoing "Thank You" to you, seriously .

Thanks for the further update. Also, thanks for the compliments. I'm happiest being of aid to others.

Postby smicha » Tue Sep 01, 2015 9:09 pm
It's high time to convince Otoy to increase the 12 gpu limit to...? 16? 20?

Smicha, do you think there is a methodology behind the limit?
I mean okay, it would have been easy to say, "Windows only allows 7-8 GPU anyway," but putting that type of idea aside now, is there an inability, do you think, on the part of the source code to support larger than 12?
Or would it likely be a business decision?

I'm sorry Smicha for poaching on a few of your opportunities to respond but I won't touch the Seinfeld/OctaneRender 3 Demo question.

I believe that the 12 GPU limit is a bit of both, but probably more heavily weighted as a business decision influenced by cloud ambitions. But also note that other GPU 3d rendering software can't scale equally, infinitely.

... .
But seriously, what is that GRID K520?

"NVIDIA® Kepler™ architecture-based GRID [GPU] boards deliver the highest-density and highest-performance solutions available for cloud gaming platforms. - See more at: http://www.nvidia.com/object/cloud-gami ... IIesH.dpuf ." [ http://www.nvidia.com/object/cloud-gami ... oards.html ].

[smicha]

Great knowledge over this topic. Thank you guys so much for sharing your thoughts and experience.

BTW I personally think that the 12-gpu limit is strictly due to a business model (especially it's valid over-the-network). Hope OTOY reads this post and will have mercy on us