Best Practices For Building A Multiple GPU System

[itou31]

Hi there,

Back to this thread as I don't know where to post.
As we know, multi-gpu need some registry editing (ex : DisplayLessPolicy, LimitVideoPresentSources) or playing with TDR.
All is good on my 4GPU rig and the bigger 7GPU rig until .... OR4 test release (and also 3.08 RC2) that need CUDA 9.10 with new drivers.
Results --> windows crash with new drivers from 387.xx to 388.xx on my 4GPU rig (P9X79 Asus Deluxe 780Ti + 3 Titan). And the last stable are 385.69.
Just tried to update with 391.24, but only 780Ti is usable. the titan are all with ex-mark in device manager.
On the 7GPU rig, no crash but near same problem. one ex-mark on one 980Ti (X99-E WS/USB3.1 + 2 1080Ti + 3 980Ti + 2 TitanBlack). Also try to playing with registry but still not work (only 6 GPU usable). Also tried to free some I/O by disable maximum add-on device on board. I'm still on win8.1, but have test last win10 1709, but same issue on the 7GPU rig.

Do someone have any idea ? Thanks

[smicha]

1. Backup your current system drive (todo, aomei,...)
2. Download https://www.microsoft.com/en-us/softwar ... /windows10 media creation software and download ISO file
3. Download rufus software and create MBR bootable USB stick (not GPT for UEFI)
4. Boot to bios and make sure legacy mode is enabled for MBR boot, not UEFI, reboot if neede to change settings
5. Boot override to the USB stick and clean install win10
6. Install INF drviers only for Asus chipset
7. Update windows
8. Install latest nvidia
9. Try Octane now

PS. In Bios set C states to off or at least to C0/C6. Check with cpuz how aggresive your load lince calibration is under full load (prime95) - if the voltage drop is too low set vcore manually.

[itou31]

Hi Smicha,
Thanks, already test all what you adviced.
It seams to have something related with different kernel driver after NVidia 385.69 drivers.
On win 8.1 or win 10, same issue.
With clean install and lastest drivers 391.24 with (cuda 9.10)

391.24.PNG

Test with octane 4 XB1 and 3.08RC2 OK but only 6 GPU available.

Clean drivers with DDU and the revert to 385.69 (the most recent that work with 7 GPU) but only cuda 9.0 and so not compatible with 3.08RC2 or the new OR 4 XB1

385.69.PNG

Don't know what to do more or stay with only 6 GPU.

[smicha]

That's something new. Can you run again clean install of new Nvidia driver?

[itou31]

Hum, still have the same problem after a clean reinstall of 391.24 drivers.
By checking device manager by connection, I think about an issue of ressources with the PCIe bridges with the new drivers.

391.24 by connexion.PNG

Even disable (in windows) the second 108ti that not connected to the monitor, the 980Ti stays still "OFF".
With 385.69 drivers : OK
with 388.92 drivers (the first one with cuda 9.1) : NOK

Oh, perhaps, if I replace the 980ti with another 1080ti, that could works !? (same GPU)...

[smicha]

Yes but the strange thing is that it is driver related.

How about Linux?

[itou31]

linux ? , i'm not so familiar with linux.
I think that I could stay with 6 GPU for now if use with Octane 4, as AI denoiser is great !

[smicha]

there is the generic pnp monitor listed - can you disable it and reinstall the driver?

[itou31]

No monitor connected.
only using LAN remote and reinstalled drivers : no way, still not work for the 980Ti.

[itou31]

Hi,
Just try to update drivers 391.35 on Win8.1 / win10 : same problem. The 980Ti on the PCI brige with the two 1080Ti doesn't work.
I just try to install win 7, and found that I have same issue with 391.35 drivers. Roll back to the 385.69 on win 7, but the problem persist. I think that is not only a drivers problem (at least on win 7, as I have many system devices without drivers).
Just to be sure that hardware is not impacted since I try windows 7 and play with bios, I just go back to win 8.1 with 385.69 drivers and all my 7 GPU work as expected.
So, stay with octane 3.08 Test 6 and 7GPUs, or go with octane 3.08 RC1 and up with only 6GPUs ...

Edit :
I just update drivers on my 4 GPU rig and have also a "same" problem:
on the Asus P9x79 deluxe with 4 PCIe :
PCIe1 : Titan (connected to monitor)
PCIe2 : titan
PCIe3 : 780Ti
PCIe4 : Titan.


With 391.35, only 780Ti works and all Titan failed

1.PNG

and viewing by connection :

2.PNG

Tutor ? do you have an idea ? I think that if I take out the 780ti, all wil work nominally. New drivers have problem with mixing differents GPU generation !?