Best Practices For Building A Multiple GPU System

[Tutor]

OS matters much in building a maxed out multiple GPU system.

Tutor, what say you? (assume PSU and power is not a concern or limitation, and assume I have most optimal BIOS for adding GPU):

I am trying to max cards on rig from 13 to 14, but under my current arrangement I always get a d4 Out of PCI Resource error. Fine, okay...but in this arrangement my Titan X is connected to slot 1 at 16x.
Could halving my 16x connection of my Titan X to 8x alone allow me to add another card?
Or, would IO space say, "No, irrespective of how many lanes are used, I can only delegate so much address memory to so many devices. It's a device count thing, not a lane usage thing...
If you have 13 at 1x, 13 at 4x, 13 at 8x, or 13 at 16x...I don't give a F&$%...you're only getting 13...because that's all I can allocate address memory to...so F&$% off with the 14, you damn F&$%ing piece of S&*$..."

It's my understanding that IO space although separate from lane count can be affected by insufficient lanes - having insufficient lanes might be manifested as an IO space issue. Thus, having sufficient lanes is a predicate to maximizing IO space capacity. So as a result of using fewer lanes per card, there's a possibility that you may be able to run another card. But for those who can use/run linux, even Otoy believes that the sky is much more accessible - viewtopic.php?f=23&t=52863&start=20#p269871 . " This year, 20 pascal GPUs should be able to roughly match the compute power of the 110 Kepler GPU cloud demo we showed at the GTC 2013 keynote. " It would not surprise me if you could run 20 Pascals on your system (while running Linux) for a Octane bench score of over 4,000 if Octane takes full advantage of FP16 where ever possible. I don't think of this as a far off into the future pipe dream, given also Otoy's announcement of Brigade arriving in V4 next year. [See also, https://devblogs.nvidia.com/parallelfor ... -cuda-7-5/ ; half precision (FP16) is twice as fast as single precision (FP32) ] Keep in mind we're just interested in graphics; not measuring someone's heart murmur.

[Notiusweb]

It's my understanding that IO space although separate from lane count can be affected by insufficient lanes - having insufficient lanes might be manifested as an IO space issue. Thus, having sufficient lanes is a predicate to maximizing IO space capacity. So as a result of using fewer lanes per card, there's a possibility that you may be able to run another card.

Wow, thanks. I am inspired to get a couple 16x (female) to 8x (male) risers and give them a whirl. If it works, I would be willing to try running my lead Titan X at 8x.

...not measuring someone's heart murmur.

Actually, my heart jumps every time I see your posts...

Indeed you had me at

Notiusweb wrote

[Tutor]

... . Wow, thanks. I am inspired to get a couple 16x (female) to 8x (male) risers and give them a whirl. If it works, I would be willing to try running my lead Titan X at 8x.

...not measuring someone's heart murmur.

Actually, my heart jumps every time I see your posts...

And so does mine to your posts, my brother. Just remember that I used words like "might" and "possible." It seems that I'm always in the learning/speculating phase too. So let me stay after class for some special needs tutoring about your discoveries/experiences.

[Notiusweb]

Just remember that I used words like "might" and "possible." It seems that I'm always in the learning/speculating phase too. So let me stay after class for some special needs tutoring about your discoveries/experiences.

I should be able to try this out in the next few days. My idea is to extend out main Titan X on slot 1 with the 16x to 8x riser, and then add in another PCI USB 3.0 1x riser on that unused hidden slot 2 for the test card as "14th GPU".
I should find out immediately whether or not this will be workable, because either my rig will either boot to BIOS, or it will still beep and give the d4 error pre-BIOS like it does now.

(1) First I will snap shot a system restore point pre-doing anything.
(2) Then I am going to install/re-establish the main Titan X at an 8x connection by itself. Once I get that working, then I'll try the rig with the other 13.

My hope is that if I can get past BIOS, I can work OS into granting me access to a full 14.

If it fails, I run only with the Titan X and system restore to before I tried, that way it's ready to still go with the other 12 like it is now.
Then I'll lay the 8x cards away for the future 2nd rig.

[smicha]

Notiusweb,
Are you running all of these cards on Extreme 11?

[Tutor]

Just remember that I used words like "might" and "possible." It seems that I'm always in the learning/speculating phase too. So let me stay after class for some special needs tutoring about your discoveries/experiences.

I should be able to try this out in the next few days. My idea is to extend out main Titan X on slot 1 with the 16x to 8x riser, and then add in another PCI USB 3.0 1x riser on that unused hidden slot 2 for the test card as "14th GPU".
I should find out immediately whether or not this will be workable, because either my rig will either boot to BIOS, or it will still beep and give the d4 error pre-BIOS like it does now.

(1) First I will snap shot a system restore point pre-doing anything.
(2) Then I am going to install/re-establish the main Titan X at an 8x connection by itself. Once I get that working, then I'll try the rig with the other 13.

My hope is that if I can get past BIOS, I can work OS into granting me access to a full 14.

If it fails, I run only with the Titan X and system restore to before I tried, that way it's ready to still go with the other 12 like it is now.
Then I'll lay the 8x cards away for the future 2nd rig.

I have a strong belief that your Asrock Extreme 11 will support many more than 13 GPUs, but from my experience it's Windows OS that restricts the IO load, even when the system could go higher. My experience on my X9DRX is about the same as your's under Windows (14 vs.13 GPU processors), but under Linux the X9DRX supports many more GPU processors. Additionally, Otoy appears to believe that Linux is the best OS for building a multiple GPU 3d rendering system - viewtopic.php?f=23&t=52863&start=20#p269871 . I know that many Octane users use 3d applications and Octane Plugins for which there is no easy install-then-render Linux support route*/. For those whose 3d applications/Octane plugins support Linux [presently Blender (free), Houdini, Maya (& Linux support recently added but not yet Octane supported - Mudbox and Motionbuilder) and Modo fit this category], Linux may be a better route to go for those desiring a self-owned, power rendering system even for those users whose systems have fewer PCIe slots than does the Extreme 11. Dual system OSes are even an option, but the GPU count above Windows' 13/14 GPU processor limit might better be integrated with ease of disconnection in mind, e.g., on an Amfeltec cluster so that they can be easily excluded from the boot process when booting into Windows for those whose systems will not boot with the excess no. of GPUs.

*/ See, e.g., http://www.pcworld.com/article/2900497/ ... -know.html .

[Notiusweb]

I tried the 14th GPU with the 8x connector, and no good. I still got the d4 error one I got to 14 GPU...It's a pre-BIOS error code, so there really isn't any negotiation except to physically move hardware or deactivate hardware from the BIOS before attempting a 14 GPU core boot. I don't think there are any hardware changes I could make to grant me more IO space.
Well, I knew it was a long shot.

I am still using the ASRock Extreme 11. ASRock had told me a while back that they don't put a restriction on the # of GPU, but there is a limited amount of board memory to allocate for connected devices. I think, from my research, I may have exploited all there is to exploit.

My only thought now is that there is one more BIOS, the most recent, which added more support for PCI connected SSD drives. Problem is it does not work with Titan X at the helm. So, I just wrote in a support ticket asking for Titan X support on the most recent BIOS. If I get that, I can explore if it will allow rig to boot to BIOS with 14 GPU connected.

In a word, CATS...at the Winter Garden theater...

[Seekerfinder]

I have a strong belief that your Asrock Extreme 11 will support many more than 13 GPUs, but from my experience it's Windows OS that restricts the IO load, even when the system could go higher.

It seems as though Asrock trumps Asus big time when it comes to PCIe lane management. I seem to hear only bad things about Asus boards (at least single cpu) with 6+ GPU setups and only good things about Asrock.
The previous gen of the Extreme 11 had 64 lanes dedicated to PCIe's. The current gen has less but it still uses the PEX 8747 chips, which seems to be magical when it comes to lane propagation.

This board is on the shortlist right now: http://www.asrock.com/mb/Intel/X99%20WS-E/

Seeker

[Seekerfinder]

Linux may be a better route to go for those desiring a self-owned, power rendering system even for those users whose systems have fewer PCIe slots than does the Extreme 11.

True, by all accounts... Does anyone know if it would be possible to use Octane with a networked GPU station on Linux while the main PC is a Windows machine? Does it blend?

Seeker

[Tutor]

I have a strong belief that your Asrock Extreme 11 will support many more than 13 GPUs, but from my experience it's Windows OS that restricts the IO load, even when the system could go higher.

It seems as though Asrock trumps Asus big time when it comes to PCIe lane management. I seem to hear only bad things about Asus boards (at least single cpu) with 6+ GPU setups and only good things about Asrock.
The previous gen of the Extreme 11 had 64 lanes dedicated to PCIe's. The current gen has less but it still uses the PEX 8747 chips, which seems to be magical when it comes to lane propagation.

This board is on the shortlist right now: http://www.asrock.com/mb/Intel/X99%20WS-E/

Seeker

That Asrock board on your shortlist has very good specs.

Linux may be a better route to go for those desiring a self-owned, power rendering system even for those users whose systems have fewer PCIe slots than does the Extreme 11.

True, by all accounts... Does anyone know if it would be possible to use Octane with a networked GPU station on Linux while the main PC is a Windows machine? Does it blend?

Seeker

I have no doubt that it would be possible to use Octane with a networked GPU station on Linux while the primary PC is a Windows OS machine; "Yes, it can be blended," but how and at what costs? I have multiple OSes (including Linux) installed on 20 of my current 24 systems (my four MacPros run only OSX and Windows). Those 20 systems are all self-builds. I'm in the process of learning the blend recipe and it's associated parts and their costs. But I don't doubt that is can be done because that is like what Otoy is essentially doing with the cloud - allowing users running Windows only systems and those with Windows and OSX (Mac) systems and those (like me) with Windows, Mac OS and Linux systems - to all be able to use the cloud (Linux OS powered) even in a render assistive capacity. Digesting this - http://developer90.openinventor.com/con ... ering-work - in depth is the the first order of learning that I've bitten off to chew. Getting infiniband boxes, cards, and cables will very likely be required depending on the size/nature/requirements of one's projects and network traffic, as well as staying compliant with the personal use restriction which is a part of the Octane license and the new 20 GPU per frame?/license limit. My goal is to use my seven Octane Stand-alone licenses on seven systems each with 20 GPU processors. For projects beyond that studio capability of 140 GPU processors I'll use the Cloud assistively when needed. My remaining 17 systems would then be dedicated to mainly CPU 3d rendering/video/video-effects rendering.