Best Practices For Building A Multiple GPU System

Discuss anything you like on this forum.
Post Reply
User avatar
Notiusweb
Licensed Customer
Posts: 1285
Joined: Mon Nov 10, 2014 4:51 am

smicha wrote:
itou31 wrote:Hi,
I finally fix the problem, so now I can use 3.08 RC4 and test the 4.00 XB1 with all GPUs ! :D
Sorry, it was completly due to my overclocking mod on the titan (on the 4GPU rig) and on the 7GPU rig, the 980Ti that I bought is also vbios mod.
I revert back to original vbios on titan and 980Ti, and now the update to 387.92 drivers works fine (enough to have cuda 9.1).
So all drivers after 385.69 fail with power vbios mod.

Thanks for sharing.
Nice! That was my experience too...
OC card's Bios seem to confuse the registry sometimes when it tries to configure the layout with a new driver.

(The only other step I would have suggested to the 'last resort' solution would have been to remove all registry entries 1st, and then go about adding each one back in. I should have added also that you would do this all after updating the main GPU to the driver you wanted.)

But!...
Well done, glad to hear that you un-F#%*'d the rig!
Win 10 Pro 64, Xeon E5-2687W v2 (8x 3.40GHz), G.Skill 64 GB DDR3-2400, ASRock X79 Extreme 11
Mobo: 1 Titan RTX, 1 Titan Xp
External: 6 Titan X Pascal, 2 GTX Titan X
Plugs: Enterprise
User avatar
Tutor
Licensed Customer
Posts: 531
Joined: Tue Nov 20, 2012 2:57 pm
Location: Suburb of Birmingham, AL - Home of the Birmingham Civil Rights Institute

For the ultimate Windows 10 machines (systems with no more than 2 CPUs), take a look, e.g., at these posts/threads:


1) https://forums.servethehome.com/index.p ... 709.17184/ &
2) https://www.reddit.com/r/NiceHash/comme ... gpu_limit/ &
3) https://answers.microsoft.com/en-us/win ... 37dfa8dec6 .

Does that bring smiles to your faces?


P.S. For the time being, I'd look to the miners for the potential of GPU maximization under Windows. Didn't the miners buy all of the top of the line, new GPUs of late? Personally, I'm sticking with Linux Mint & Ubuntu Studio for most of my GPU centric non-Apple systems and, of course, MacOS for my MacPros.
Because I have 180+ GPU processers in 16 tweaked/multiOS systems - Character limit prevents detailed stats.
User avatar
Notiusweb
Licensed Customer
Posts: 1285
Joined: Mon Nov 10, 2014 4:51 am

I actually have a question...I know, it has been a long time!

I changed my PC case which has given me larger room at bottom, and it has me considering moving a whole GPU card off a splitter, which is connected 1x, into the main case, so that I can have 2 directly connected GPUs on the mobo.
I would stick the 2nd card in the 7th slot, since it will have much breathing room now over the PSU below it.

But my worry is that in doing so, I will then inadvertently limit the amount of GPUs I can otherwise run off of the external 1x splitters connections, because there will be more lane usage by the 2nd GPU @8x direct on the mobo.
(not to mention power usage on the mobo, but I am imagining this will not be a factor as long as my mobo PSU is strong enough).

Before I start messing with the rig, I figured I should ask if:
(A) yes, adding a 2nd card direct to mobo will in fact cut into the amount of 1x connections that can be otherwise be handled off of the other PCI slots with only 1 GPU direct to mobo (in which case my max GPUs will be cut down), or
(B) no, adding a card direct to mobo will only cause the mobo to then split the actual PCI-lanes (not slots) in order to distribute the usage, and the 1x connections will remain intact

Any thoughts?
Thanks, and hope all is well!
Win 10 Pro 64, Xeon E5-2687W v2 (8x 3.40GHz), G.Skill 64 GB DDR3-2400, ASRock X79 Extreme 11
Mobo: 1 Titan RTX, 1 Titan Xp
External: 6 Titan X Pascal, 2 GTX Titan X
Plugs: Enterprise
User avatar
Tutor
Licensed Customer
Posts: 531
Joined: Tue Nov 20, 2012 2:57 pm
Location: Suburb of Birmingham, AL - Home of the Birmingham Civil Rights Institute

Notiusweb wrote:I actually have a question...I know, it has been a long time!

I changed my PC case which has given me larger room at bottom, and it has me considering moving a whole GPU card off a splitter, which is connected 1x, into the main case, so that I can have 2 directly connected GPUs on the mobo.
I would stick the 2nd card in the 7th slot, since it will have much breathing room now over the PSU below it.

But my worry is that in doing so, I will then inadvertently limit the amount of GPUs I can otherwise run off of the external 1x splitters connections, because there will be more lane usage by the 2nd GPU @8x direct on the mobo.
... .

Any thoughts?
Thanks, and hope all is well!
Hello Notiusweb,

Presently, what make and model of

(1) Motherboard,
(2) CPU(s & No. of them),
(3) Splitter (& No. of them and connection points on each one of them) and
(4) GPU(s & No. of them)

are we dealing with in/for that system?

N.B. - The impact of that no. that follows the little "x" can be a little deceiving for GPU rendering. For example, with Octane an x8 PCIe connection can be faster than an x16 connection. See, e.g., https://www.pugetsystems.com/labs/artic ... 3_0x8vsx16 . That's a little part of the reason why I'm fully satisfied with my six SuperMicro X9DRX's - each with 10 x8 slots and 1 x4 slot. While specs do matter, how they matter can be a bit surprising. Imagine trying to throw a bowling ball strike on a 100 yard wide lane. Sometimes for accuracy, it's better to be constrained. That's also why I need to look in depth/detail regarding what you have to better answer your question.
Because I have 180+ GPU processers in 16 tweaked/multiOS systems - Character limit prevents detailed stats.
User avatar
mrpinoux
Licensed Customer
Posts: 59
Joined: Tue Sep 30, 2014 8:51 pm
Location: Los Angeles
Contact:

Hi community!

Currently rebuilding my rig of 4 x gtx 780 watercooled on ASUS X99 E-WS inside 900D made 3 years ago with the help of tom and smisha, with 4 x 1080ti's (to maybe seven?) with the same Mother Board inside the termaltake X9.
I have a 1600 Power supply. All the 1080tis are 1 slot mod.

Is there anything specific to know or to do, or not to do if I want to get 7x1080tis with the X99 E-WS? I know that i need more power and I also have another 1200 Power supply and a cable to connect them together.
I read somewhere about activating the 4G decoding in the BIOS in order to get WIN 10 to use all the card. Is this still relevant? Also something about C State, any input on this?

Is it enought with the 1200 to get the 3 more 1080ti, or should i just plug the 780's or get a new power supply? Also, fyi, I have 1 x 480 radiator, 1 x 360 and 1 x 240, shoud I replace one of them, or add one more for a 7 x gpu rig?

Should I stick with a 4 x gpu rig or is it worth to get the 7 in it?

Thanks!
Win 8.1 64 | Geforce GTX780 x 4 | i7 5960X | 64GB
User avatar
Tutor
Licensed Customer
Posts: 531
Joined: Tue Nov 20, 2012 2:57 pm
Location: Suburb of Birmingham, AL - Home of the Birmingham Civil Rights Institute

mrpinoux wrote:Hi community!

Currently rebuilding my rig of 4 x gtx 780 watercooled on ASUS X99 E-WS inside 900D made 3 years ago with the help of tom and smisha, with 4 x 1080ti's (to maybe seven?) with the same Mother Board inside the termaltake X9.
I have a 1600 Power supply. All the 1080tis are 1 slot mod.

Is there anything specific to know or to do, or not to do if I want to get 7x1080tis with the X99 E-WS? I know that i need more power and I also have another 1200 Power supply and a cable to connect them together.
I read somewhere about activating the 4G decoding in the BIOS in order to get WIN 10 to use all the card. Is this still relevant? Also something about C State, any input on this?
.

I recommend that you enable "above 4G " in your system's bios settings. You appear to have nothing to lose and potentially gain a lot. I recommend that you discuss C-state adjustment (and your question, below, regarding water cooling) with Smicha.
mrpinoux wrote: Is it enought with the 1200 to get the 3 more 1080ti, or should i just plug the 780's or get a new power supply? Also, fyi, I have 1 x 480 radiator, 1 x 360 and 1 x 240, shoud I replace one of them, or add one more for a 7 x gpu rig?

Generally, for GPUs that only render in octane, ensure that the power supplied to those GPUs is at least 85% of spec for each and all GPU(s). So you need to be guided by specification(s) of the particular GPUs, as well as power needs of other things in your system that contribute to power needed. Also note that different GPU makes have different power specs. However, my guess is that the addition of using the 1,200 watt PSU (along with the 1,600 watt PSU) will be enough for 7 GPUs.
mrpinoux wrote: Should I stick with a 4 x gpu rig or is it worth to get the 7 in it?
.

Since I have no idea what you do with your system, or what your needs are, I cannot answer this question for you. However, and as only an example - generally, a 3d animator and/or video/movie maker needs more (in quantity) and more powerful GPUs than one who produces only stills. But even in that scenerio, variables such as output size/aspect ratio also matter.
Because I have 180+ GPU processers in 16 tweaked/multiOS systems - Character limit prevents detailed stats.
User avatar
Notiusweb
Licensed Customer
Posts: 1285
Joined: Mon Nov 10, 2014 4:51 am

Thanks Tutor....
Okay, well I figured out in my 'case' (get it!?...) that putting a 2nd GPU on the mobo was just fine, I still was allowed to run as many max GPUs as before.

But I experienced something else more unfortunate...And it matters not where I had the GPUs
Win 10 updates after 17##, like the 18## series, are bugged with not allowing more than 8 GPUs. Many bitcoin miners have noted this in online forums, and have back-dated to the 17## series of OS updates in order to run their full GPU gamut.
In my case though, I only installed after the 18## series of updates was released, meaning I can't back-date to a 17## series update, so I would have to gain possession of the 17## update, then reinstall windows 10 and manually setting it to the latest 17## series of OS update. And I would have to reinstall all of my stuff....UGH....

I do not know why this problem exists for the 18## series...
My BIOS, for example, lets all my cards through, but no matter what old-school registry Dword values I input, or drivers I use, or Device manager finagling I do, I cannot get Win 10 to recognize a working 8+ GPUs.
I tried a bazillion different things - increasing Virtual memory paging file size, searching for IRQ conflicts, disabling things in BIOS, as I know this stuff pretty good...
like, I've been around the block a few times in my day with this stuff, you know?....
And, if I run the GPUs in separate instances, they all work, as long as < 9. Furthermore, if I run my Win 7 disk instead as the booting drive, I can run all of my GPUs, so it is totally a Win 10 thing.
The old registry tricks etc that worked in Win 7 do not work in Win 10.
Annoying, what a shame. I hope a later update fixes it again.
I actually posted about it on a Microsoft forum, and I got the suggestion to backdate my OS to Win 10 17.03 or 17.09.
:(
Win 10 Pro 64, Xeon E5-2687W v2 (8x 3.40GHz), G.Skill 64 GB DDR3-2400, ASRock X79 Extreme 11
Mobo: 1 Titan RTX, 1 Titan Xp
External: 6 Titan X Pascal, 2 GTX Titan X
Plugs: Enterprise
User avatar
Notiusweb
Licensed Customer
Posts: 1285
Joined: Mon Nov 10, 2014 4:51 am

So I spoketh too soon. I was comparing my:
(a) Win 7 64 - 1 Titan X (on mobo) + 12 Titan Z (external),

to my new setup on Win 10 machine, which is now:
(b) Win 10 64 Pro - 1 Titan X (Mobo main) and 1 Titan Xp (Mobo 2nd) + 6 Titan X Pascal (external)

I was saying 13 vs 8, how come the limit.

But now I tried with Win 7 64, the 'b' setup, and it too, only allows 8. Which is so strange, because I am using the same exact Win 7 drive and following all the rules that allowed me to get 12 Titan Z on this same OS routine.
Now, the motherboard BIOS lets it all of thecards I want in, it's just getting Win 10, and now also Win 7, to recognize, is the issue.

This leads me to believe that one of the following may be the issue:

(1) Maybe the 12 Titan Z was still really only 6 Gpu cards for the OS? Although, it was always listed everywhere as 12 separate instances of a GPU core, but maybe it really was 6 in the Win 7 sceheme? the 12 Titan Z was 6 physical PCI connections, not 12. Now, 3 were attached as 6-cores to Mobo per a splitter, at 1 PCI slot entry point hardware-wise. But, the card slots themselves, attached to the splitters, always were 6. Now I have 8.
So maybe 8+ is too many physical PCI-card-specific connections or something?
(2) Maybe the altered PCI slot setup is really now affecting the # of cards Win 7 would recognize, but it seems like the BIOS would be the one with the issue if anything hardware, you know?
(3) Maybe these later cards (Titan X Pascal, Xp) use more memory resource or something, which makes the OS say, "Nah, I can only handle 8...."

Also, I know Smicha has 11 separate physical connections, but was Linux, not Windows.
The Bitcoin miners also vaguely refer to their cards, a lot of times it's half Radeon and half Nvidia, maybe there is a reason they had to do this.
Win 10 Pro 64, Xeon E5-2687W v2 (8x 3.40GHz), G.Skill 64 GB DDR3-2400, ASRock X79 Extreme 11
Mobo: 1 Titan RTX, 1 Titan Xp
External: 6 Titan X Pascal, 2 GTX Titan X
Plugs: Enterprise
User avatar
Tutor
Licensed Customer
Posts: 531
Joined: Tue Nov 20, 2012 2:57 pm
Location: Suburb of Birmingham, AL - Home of the Birmingham Civil Rights Institute

There are reasons for each of your observations. My experiences have lulled me into making these estimations (some old, others new):

1. CURRENTLY, Linux (free) is the best/stable OS for systems built to house eight or more GPU processors. If I had been asked "which distro" two weeks ago, I would have then responded - "Ubuntu Studio and Linux Mint." However, I'm studying what appears to be a real challenger for the crown - whimsically called "PuppyLinux." [See http://puppylinux.com & http://puppylinux.org/main/Overview%20a ... tarted.htm ] Windows (not free) is best for systems built to house less than eight GPU processors if one wants to overclock them. Windows systems are more easily overclocked by the free overclocking utilities that are available. Notice that I use the term GPU processors because, like you, I also have dual processor cards such as GTX Titan Zs, GTX 590s and a GTX 690 [such cards can be designed to handle IO space issues more efficiently because the two GPU processors share almost all other card components]. Also, different OS versions can make/break the or/deal. In the situation you describe, is it ordeal (without the slash)? Luckily, Linux has been supporting the top GPU maxed out systems long before Windows 7-10 were announced.

2. Newer GPUs tend to use more input/output (IO) space than their predecessors. This trend I project will continue. New/additional/improved features lures us to buy them. Those features can come at a hidden cost to those in need of running more many GPU processors in a single system - they require more & more IO space. The upside is that "Newer GPU Processor" generally means "Faster Rendering GPU Processor."

3. Nvidia erects IO caverns in its products a lot differently than AMD does in its products. That difference can be a good thing - that means than even when a system lets one know that it's IO space for GTXs has been tapped out, usually there's still a lot of empty IO space to support some AMD GPUs. So, if one also has a need for OCL rendering, there's room enough to make CUDA and OCL rendering occur simultaneously with a bunch of AMD and Nvidia GPU processors. I've been mixing them for years. But I do have many, many more GTXs than I have AMDs.

4. Different motherboards support different amounts of IO space. When comparing motherboards, note the difference in the number of visible PCI-e slots. Generally a good indicator of which motherboard will support the the greater number of GPU processors is the one with the most PCI-e slots. The number of visible slots isn't necessarily the max, but I suggest most often that the motherboard with more such slots is more likely to support even more GPUs than the comparator *1/].

5. Generally, the more CPUs a system has, the greater the number of GPU processors that can be supported. Why? A two CPU system generally has the potential to handle more IO activity than a single CPU system has. Moreover, a quad CPU system generally has the potential to handle more IO activity than a dual CPU system has, etc. etc. I still recommended *2/ ],that we completely digest the graphical depiction of the target system's motherboard layout. How many "words" can a picture be worth?

6. Regarding your last post, try to give me each and every change that you made to each component of the system, when the change occurred and what OS was used at what period in time. Are (or were) you using Amfeltec components and/or other splitters/other special connectors, and if so, which ones and has there been a change involving any of them, the nature of the change and when did the change occur? Frankly, I'm not surprised that you cannot now run more GPUs in the Windows version that you now have. What would be surprising is that you hadn't made any changes to the system and could run six Titan Z cards in the past under Windows pre-version 10 and not Server.

*1/ I stand on the shoulders of the greatest pioneer. Back on a day in late 2013 - that great pioneer - zzz1000 posted a topic entitled - "18 GPUs in a single rig and it works." [ https://devtalk.nvidia.com/default/topi ... -it-works/ ]. I've read that topic (and others of his posts elsewhere) many times since I was blessed to find them and on each occasion I've said, "How did I miss that next point that was revealed so subtilely by his recorded observations?" Note what motherboard he says that he used and what motherboard model that I currently have seven of. Also, note who zzz1000 was inspired by, then study the posts regarding FASTRA.

*2/ [ viewtopic.php?f=40&t=43597&start=200#p241271 ]
Because I have 180+ GPU processers in 16 tweaked/multiOS systems - Character limit prevents detailed stats.
User avatar
Notiusweb
Licensed Customer
Posts: 1285
Joined: Mon Nov 10, 2014 4:51 am

Hi Tutor, I didn't say before, but good to hear from you again my friend, I hope all is well with you!

I have been now exploring and playing with different PCI entry points, and it just does not want to go beyond 8 while using the later Pascal cards.
I actually today tried with 3 Titan Z for 6 cores, and the Titan Z's were added in and installed fine. So I had 2 mobo Titan X (X and Xp) and the 3 Titan Z, for atotal of 8.
Then I powered up 3 of the external Pascals Titan X, and booted for a total 11.
And.... it failed to recognize 3 cores from Titan Z (as in 1.5 cards-worth of Titan Z physical GPU), pulling me down to the 8 again.

So I looked in registry editor, and low and behold the were were suddenly now 3 Titan Z that had no LimitPresentSources, or whatever, RegD word setting!
So I got excited as it mathematically matched up to the uninstalled 3 and I added in the entries and rebooted.

Ugh...But then it just seemed to change the Windows Device Manager error code from a Code 35 to a Code 12, and now it just can only see the 8.

All Amfeltec, this really leads me to believe, like you were explaining, that the Pascal series is taking more resources from the OS than the earlier Titan Z.
Interesting, the mobo always lets as many as 13 in as long as enough peripheral devices are disabled in BIOS.
I imagine now it just so happened to be the perfect arrangement for the ASrock X79 Extreme11 - 12 Titan Z + 1 Titan X.
But, that used 6 GB Vram, and Out of Core runs slower than a real 12 GB Vram card combo, like the 8 Titan X bunch I can alternatively use.

So, where I am now, is this:
I am maxed out as far as GPU speed goes then.
If I want faster, I have to get Volta!
Only this might pull me down to 7, or 6.....which maybe nets me to the same, or nudge better, performance after all...F#$%!
Win 10 Pro 64, Xeon E5-2687W v2 (8x 3.40GHz), G.Skill 64 GB DDR3-2400, ASRock X79 Extreme 11
Mobo: 1 Titan RTX, 1 Titan Xp
External: 6 Titan X Pascal, 2 GTX Titan X
Plugs: Enterprise
Post Reply

Return to “Off Topic Forum”