Best Practices For Building A Multiple GPU System

[smicha]

Itou,

I've heard that there is a software (?) that fixes vram issue for Titans only on win10. So maybe this is new driver related. I'll test today 11 GPU 1080ti machine with win10 and 391 and will let you know if all works fine.

[itou31]

Thanks Smicha,
Do you know the name of this software or where you see it ?
I think that your 11 GPU 1080Ti will be fine with the new drivers 391, as they are same generation.

[Notiusweb]

Oh my goodness...I missed these discussions!

Hope everyone is well, I have been messing with PBR and Unreal Engine. Okay, back to this...

itou31, I had an issue once like this after installing and then reverting Nvidia drivers. Are any of the cards in your rig OC, like is there one 980Ti that is not OC and then the other one is? Sometimes the profile in regedit is what is getting screwy, not necessarily the driver.
I have a hunch that if you erased the registry for all cards but the main mobo GPU card,

2) Navigate to the following key:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Class\{4D36E968-E325-11CE-BFC1-08002BE10318}

3) You will see keys for each video card starting with "0000" and then "0001", etc.


and then installed each card back in one by one, it would re-establish a working profile. Sometimes when doing it all at once, the registry is like, "Sorry 980Ti-B, this slot is for 980Ti-A, not 980Ti-B", meanwhile "A" is sitting in "B's" slot, but working. And like a Catch-22, you can't install it.

I guess if worse comes to worse you could try this, it is not at all guaranteed. But logically, it would allow a more systematic and progressive troubleshooting.
Sorry you are experiencing this....
I'll say it for you....
F&%$ !

[itou31]

Hi Notiusweb,
welcome back to this thread !
No, I don't want to add one by one : I'm in water cooling with 7 GPUs in one block.
I could try the registry.
But when I do a fresh install (so new registry) I have the same issue with the new drivers (tests all whql from 387.92 --> 391.35, with a DDU each time.)
I'm ready to stay only with 6 GPU with octane 4, and wait for 1080ti gets cheaper.

[Notiusweb]

Hi Notiusweb,
welcome back to this thread !
No, I don't want to add one by one : I'm in water cooling with 7 GPUs in one block.
I could try the registry.
But when I do a fresh install (so new registry) I have the same issue with the new drivers (tests all whql from 387.92 --> 391.35, with a DDU each time.)
I'm ready to stay only with 6 GPU with octane 4, and wait for 1080ti gets cheaper.

What if you only unplug the power cables from the PSU, but not from the cards themselves, and then run the rig.
You wouldn't have to unplug anything PCIE-wise or Water-tubing wise. You would just have the cards unplugged as inactive, and you could install one by one:

1) You would power off, plugin in card 1's cables to the PSU, power on and install card 1,
2) then power off, plugin card 2's cables to the PSU, power on and install card 2, with card 1 running
3) then power off, plugin card 3's cables to the PSU, power on and install card 3, with cards 1 and 2 running,
Etc...

That's how I did it in troubleshooting.

Oh....On the Octane 4 side, yeah, it will be interesting to see what power the more developed uber-material Denoising Brigade machine can do!

[itou31]

Oh, thanks for the advice.
I don't know that way, the card is disable. I will try that.

[smicha]

Guys,

Has anyone managed to pass-through from Linux (Ubuntu, Mint,....) a PCI device (GPU) into a virtual machine, to Windows?

[itou31]

Hi Notiusweb,

I cannot manage to disable GPU by disconnecting power. They are still there in windows and have same issue with 391.35 drivers.
I think that I the fault is 780ti between the 3 Titans. So still stay with 385.69 drivers that works with all GPUs....

[itou31]

Hi,
I finally fix the problem, so now I can use 3.08 RC4 and test the 4.00 XB1 with all GPUs !
Sorry, it was completly due to my overclocking mod on the titan (on the 4GPU rig) and on the 7GPU rig, the 980Ti that I bought is also vbios mod.
I revert back to original vbios on titan and 980Ti, and now the update to 387.92 drivers works fine (enough to have cuda 9.1).
So all drivers after 385.69 fail with power vbios mod.

[smicha]

Hi,
I finally fix the problem, so now I can use 3.08 RC4 and test the 4.00 XB1 with all GPUs !
Sorry, it was completly due to my overclocking mod on the titan (on the 4GPU rig) and on the 7GPU rig, the 980Ti that I bought is also vbios mod.
I revert back to original vbios on titan and 980Ti, and now the update to 387.92 drivers works fine (enough to have cuda 9.1).
So all drivers after 385.69 fail with power vbios mod.

Thanks for sharing.