Best Practices For Building A Multiple GPU System

[smicha]

Guys,

I am preparing for Monday works on registry for 7 gpus. I just would like to make sure I understand your advice correctly. What I see in my personal regedit (4 gpu machine) the gpu that handles a monitor has NVSPCAPS entry, contrary to all other 3 gpus. Does it also appear in your registry?

So my question is if these two scenarios differ?

Scenario #1
1. I shall leave only 1st gpu powered by cables (I need to keep all 7 in pcie slots due to water)
2. Run DDU. Shall I see clean registry with 0000 only when I run it?
3. Install new clean nvidia driver with 1st gpu -> reboot -> connect power cables to the 2nd card (3pcie slot) -> clean install of nvidia -> reboot -> power the 3rd gpu and so on...

or Scenario #2
1. Same as #1
2. Enter regedit and delete all visible 0000, 0001,... entries, leaving the main 4D3... folder and reboot
3. Same as #1

Is this correct?

When exactly a next registry entry (0001) is being created - when I install nvidia driver? Or when I power a gpu with PSU cables and reboot?

Thank you again so much for your help.

[Seekerfinder]

Guys,

I am preparing for Monday works on registry for 7 gpus. I just would like to make sure I understand your advice correctly. What I see in my personal regedit (4 gpu machine) the gpu that handles a monitor has NVSPCAPS entry, contrary to all other 3 gpus. Does it also appear in your registry?

So my question is if these two scenarios differ?

Scenario #1
1. I shall leave only 1st gpu powered by cables (I need to keep all 7 in pcie slots due to water)
2. Run DDU. Shall I see clean registry with 0000 only when I run it?
3. Install new clean nvidia driver with 1st gpu -> reboot -> connect power cables to the 2nd card (3pcie slot) -> clean install of nvidia -> reboot -> power the 3rd gpu and so on...

or Scenario #2
1. Same as #1
2. Enter regedit and delete all visible 0000, 0001,... entries, leaving the main 4D3... folder and reboot
3. Same as #1

Is this correct?

When exactly a next registry entry (0001) is being created - when I install nvidia driver? Or when I power a gpu with PSU cables and reboot?

Thank you again so much for your help.

Smicha,
I'm speculating here and I hope Tutor or someone offers you better news but from my experience it seems the reg entry is created when the card is first detected in the slot. It allocates i/o space at this point and issues a card ID (this is where the 4G decoding on multi-card setups comes in). It then checks if aux power is needed for the specific card type - if this test is not satisfied, problems ensue.

I would be very interested to see how this goes. In my experience the physical removal and systematic replacement of cards is important when GPU issues like this arises. I am not sure the bios will like it if you leave cards in the electrical connection on the PCIe and simply unplug the aux cables. This is the challenge with 7-card on-board water-cooled systems and PCIe-acrobatics - particularly on single CPU system.

It's a shame that companies like Asus don't wake up to the high-end multi-card requirement of modern GPGPU applications and offer a custom BIOS solution.

Good luck and I hope you find a solution quickly!

Seeker

[smicha]

Seeker,

Thank you. Here is my point on this case:

1. 4G decoding enabled is a must for new bios from Asus. Or at least the mobo must detect 7 gpus and enable it in bios as defaults. Otherwise the mobo will not boot to bios (luckily this is not the case for now). So if someone will reset bios with 7 gpus on the mobo he's is in a very inconvenient situation: gpus must be removed entirely from the mobo and 4g must be enabled again. This is where Asus really sucks - they knew about it since the Linus 7 gpu build and since then no update has been released.
2. Asus must also correct the bios in such a way that when it is flashed with 7 gpus the mobo/bios does not go down giving 00 code with missing James Bond and his lucky number 7
3. Regedit - as I think - is a windows bug and whenever we run clean install of nvidia drinver all registry shall be restored or at least working flawlessly.

I'll report how it goes guys tomorrow morning.

[Notiusweb]

Smicha, was there data on the drives that you were looking to preserve, otherwise it seems you could just reinstall OS and replay your steps when you installed GPUs last time without the registry fuss?

[Tutor]

Guys,

I am preparing for Monday works on registry for 7 gpus. I just would like to make sure I understand your advice correctly. What I see in my personal regedit (4 gpu machine) the gpu that handles a monitor has NVSPCAPS entry, contrary to all other 3 gpus. Does it also appear in your registry?

So my question is if these two scenarios differ?

Scenario #1
1. I shall leave only 1st gpu powered by cables (I need to keep all 7 in pcie slots due to water)
2. Run DDU. Shall I see clean registry with 0000 only when I run it?
3. Install new clean nvidia driver with 1st gpu -> reboot -> connect power cables to the 2nd card (3pcie slot) -> clean install of nvidia -> reboot -> power the 3rd gpu and so on...

or Scenario #2
1. Same as #1
2. Enter regedit and delete all visible 0000, 0001,... entries, leaving the main 4D3... folder and reboot
3. Same as #1

Is this correct?

When exactly a next registry entry (0001) is being created - when I install nvidia driver? Or when I power a gpu with PSU cables and reboot?

Thank you again so much for your help.

Smicha,

I've yet to use DDU, so I cannot offer any advice regarding it. But when one who has only one GPU and first installs it into a new system and boots up for the first time, registry entry 0000 is recreated (for just a plain/generic VGA card). To be sure, that registry entry gets modified after Nvidia driver software is installed (now card attribute specificity gets applied to that registry entry). Although its been a long time since I installed more than one GPU initially into a system, I can recall, before installing Nvidia drivers, seeing more that one plain/generic VGA card listed in the registry (which is a function of Windows and thus not the Nvidia drivers), and being recognized by regedit, but I don't recall the exact circumstances. Thus, if the other cards are totally (and I emphasize "totally") not perceived at all by the system, your stratagem may work, but if the cards are at all perceived as present because of that tiny 75 watts from the PCIe connector, the other GPUs may then likely be evaluated as incorrectly working (i.e., failing) GPUs, then, you'd be S...OutOfLuck because they'd get picked up by regedit.

Smicha,
I'm speculating here and I hope Tutor or someone offers you better news but from my experience it seems the reg entry is created when the card is first detected in the slot. It allocates i/o space at this point and issues a card ID (this is where the 4G decoding on multi-card setups comes in). It then checks if aux power is needed for the specific card type - if this test is not satisfied, problems ensue.

I would be very interested to see how this goes. In my experience the physical removal and systematic replacement of cards is important when GPU issues like this arises. I am not sure the bios will like it if you leave cards in the electrical connection on the PCIe and simply unplug the aux cables. This is the challenge with 7-card on-board water-cooled systems and PCIe-acrobatics - particularly on single CPU system.

It's a shame that companies like Asus don't wake up to the high-end multi-card requirement of modern GPGPU applications and offer a custom BIOS solution.

Good luck and I hope you find a solution quickly!

Seeker

Seeker,

It's been my experience (and maybe my poorer and aged recollection) that some of the specifications that you mention in your first paragraph do not occur until one installs one of Nvidia's drivers. I liken first boot to be like a knock at your door or a ring of your doorbell on a dark night where there aren't any street lights and your porch light is out of service. If you hear those sounds, you realized someone is at the door, but you don't know whose there. The installation of Nvidia's driver is likened to your peering out the peep hole and your rebooting after installation of the driver is like the visitor saying, "This is Tutor; are you going to open the door?" The reason why I recommend enabling on first boot "above 4G" functionally is because it's like keeping you door's peephole clean, your doorbell in working order and your porch lights working. However, I don't believe, although I'm not sure, that IO space allocation occurs before Nvidia driver installation; but that uncertainty just makes me double-down and recommend that 4G decoding be initialized ASAP. I do fully agree, "... the reg entry is created when the card is first detected [by Windows] in the slot."

Smicha, was there data on the drives that you were looking to preserve, otherwise it seems you could just reinstall OS and replay your steps when you installed GPUs last time without the registry fuss?

In building a number of multiple GPU systems on Gigabyte x79s, Tyans and Supermicro , what I've found is that a best practice is to decide first which GPU will be used for interactivity/display purposes. Why? Because that's the GPU that needs to be first registered by the registry and it's best to install that GPU alone AND first. In my builds currently, that's typically a GT 640 4G. When I've tried swapping for interactivity from one GPU to another one, I've sometimes had to speed hours playing Inspector Gadget. So if I was preparing a system for someone else and their preference was to output video from a DVI port and only one card now has such a port (maybe because I'd hacked the others off), then in a clean registry environment I'd first install only the GPU with the video card with the DVI port and boot the system; stop off in bios and enable above 4G and save that configuration; reboot then next install the Nvidia and CUDA drivers for that one card and reboot and install Octane. After witnessing a fully working installation of that software, I'd reboot and edit the registry as Otoy recommends in Octane for those with cards that are unseen (BTW - this works for other 3d GPU renderers), but you say, "there's only one card there in that scenario." So, I'd respond that I prefer disease prevention to disease cure - and prevention (along the way) takes less time and effort than finding cures, particularly when I realize that I'll be shortly {and that my aim is that I'll} be pushing frontiers where the problem to be cured by the registry hack may likely occur. Thus in sum, I wouldn't exactly replay the steps that got Smicha to where he is today. I'd tailor the install to the system owner's preferred Interactivity/display GPU with the DVI port. Might not extant hardware (or the opposite, i.e, a lack of DVI ports on 6 of 7 GPUs) affect whether a DVI driver might be properly installed?

Seeker,

Thank you. Here is my point on this case:

1. 4G decoding enabled is a must for new bios from Asus. Or at least the mobo must detect 7 gpus and enable it in bios as defaults. Otherwise the mobo will not boot to bios (luckily this is not the case for now). So if someone will reset bios with 7 gpus on the mobo he's is in a very inconvenient situation: gpus must be removed entirely from the mobo and 4g must be enabled again. This is where Asus really sucks - they knew about it since the Linus 7 gpu build and since then no update has been released.
2. Asus must also correct the bios in such a way that when it is flashed with 7 gpus the mobo/bios does not go down giving 00 code with missing James Bond and his lucky number 7
3. Regedit - as I think - is a windows bug and whenever we run clean install of nvidia drinver all registry shall be restored or at least working flawlessly.

I'll report how it goes guys tomorrow morning.

Smicha,

If you are determined to correct the situation without having to unseat the other water-cooled GPUs, just remember that you should prevent the Windows generic video drivers and Nvidia's specific drivers from knowing that there are other GPUs in play until the DVI equipped GPU has been coronated king. Therefore my question to you: Is there such a thing as extremely thin non-conductive wrap that might work safely in PCIe slots.

[smicha]

Tutor,

So when I left the machine for night without power cables and turned it on there were 0000-0006 regedit items. But when I deleted them all and reinstalled nvidia driver (one by one) only 0000-0003 appeared, 1st gpu as 0002. The one with dvi port is unwanted one (7th, that was accidentally connected and messed all up).

So I clean installed win10 and nothing. Still same - only 4 devices in manager, even if I power 1st by cables. So I cannot get 7 of them in registry. So my plan for tomorrow is to unplug cables for few hours and boot the machine with all 7 gpus powered and try to run nvidia clean install. I think I will also manually change pcie gen to 2 (x8 in bios) fo all gpus (from auto).

I think that now my goal is again to see 0000-0006 in regedit.

Last option will be removing gpus with draining the loop...

[Tutor]

My misunderstanding - I thought the owner wanted to use GPU with the DVI port. That pic screams to me to use registry hack on a cleaned registry (unless other cards are DOA). Please correct me if I'm misinformed, but I thought that at some point in the past all 7 GPUs were working properly.

P.S. Here a shot into the darkness about another possibility. Could seeing 7 GPUs that are virtually all the same except just one of them has a working DVI port be a potential source of some confusion for drivers.

[smicha]

Yes - all 7 were working like a charm unless the guy connected a monitor with dvi cable to the lowest 7th card (I used only DP cable and 1st gpu).

So what exactly shall I do - registry hack? I have now access to the machine via temaviewer.

[smicha]

PS. There is only 1st and 2nd powered by cables. Is there any way to get them all (7) in device manager only from 75W pcie power, without cables?

[Tutor]

Yes - all 7 were working like a charm unless the guy connected a monitor with dvi cable to the lowest 7th card (I used only DP cable and 1st gpu).

So what exactly shall I do - registry hack? I have now access to the machine via temaviewer.

Do this first and let me know what happens.

"Windows and the Nvidia driver see all available GPU’s, but OctaneRender™ does not.

There are occasions when using more than two video cards that Windows and the Nvidia driver properly register all cards, but OctaneRender™ does not see them. This can be addressed by updating the registry. This involves adjusting critical OS files, it is not supported by the OctaneRender™ Team.

1) Start the registry editor (Start button, type “regedit” and launch it.)
2) Navigate to the following key:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Class\{4D36E968-E325-11CE-BFC1-08002BE10318}
3) You will see keys for each video card starting with “0000″ and then “0001″, etc.
4) Under each of the keys identified in 3 for each video card, add two dword values:
DisplayLessPolicy
LimitVideoPresentSources
and set each value to 1
5) Once these have been added to each of the video cards, shut down Regedit and then reboot.
6) OctaneRender™ should now see all video cards." [ https://docs.otoy.com/#sector_9 ]