Our experience (and solution) with Render Failure/Cuda error

[GIOLETS]

Our company have spent some times of last week trying to reduce or resolve problems of Render Failure and Cuda error that frequently occurred on heavvy scenes.
we have tested problematic scenes on different machines with different configuration.
in particular one scene 8probably with some materials defect) give problems on almost all the machine (is the scene that I sent you Ahmet).
Here the result of our "investigations":

1) some hardware of the machines are most strong and reliable than others. In particular these aspect are really important building a Pc:
- power supply (obviusly) choose a good one with sufficient power for all the GPU (we prefer Corsair HCI series). Check if your motherboard is provided of a supplementary PCI connector for multiple GPU installation and ALWAYS connect it!
Our Asus X99 E WS and also AsRock Z97 Extreme 9 have this kind of connector.
- cooling system: our experience tells that water cooling (where is possible to apply) is really a good choice and allow to install the GPU directly on the motherboard (other aspect that is important)
- chech if on your machine the File Paging is correclty configurated. On one machine that constantly gives Cuda error we have found that the File Paging were missed. Reconfigurated it and the cuda error goes away immediatly.

2) The most important thing, that here have resolved (at the moment) all our Render Failure problem, is to modifiy the TDR parameters. We have choosed to set TDR delay to 20 and TDR Level to 0. We think that only one should be the solution (and in all probablity is the first one), but in all the machine modifying these parameters the error goes away immediatly.
Here the details:
Type "regedit" and modifiy the Windows Register :

TdrLevel - Specifies the initial level of recovery. The default is to recover on timeout, which is represented by value 3.
KeyPath : HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\GraphicsDrivers
KeyValue : TdrLevel
ValueType : REG_DWORD
ValueData : 0 - Detection disabled OR 3 - Recover on timeout we have setted it to 0

TdrDelay - Specifies the number of seconds that the GPU can delay the preempt request from the GPU scheduler. This is effectively the timeout threshold. The default value is 2 seconds.
KeyPath : HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\GraphicsDrivers
KeyValue : TdrDelay
ValueType : REG_DWORD
ValueData : Number of seconds to delay; 2 seconds is the default we have setted it to 20

So, hoping to help other user, this is our experience.

[mattking]

Hi GIOLETS

Thanks for the info.

On a side question, I noticed you said you used the Asus X99 E WS mobo, I have that one too, what do you mean when you say "Check if your motherboard is provided of a supplementary PCI connector for multiple GPU installation" What is a supplementary PCI connector? Also, how many GPUs do you have connected to your Asus? I bought 4 GPUs but the 4th dosnt fit...because of the headers on the edge of the mobo blocking the 4th GPU from being pushed all the way in. Please let me know.

THANKS!

[GIOLETS]

It's the connector nr. 2 on the Asus Manual Page 1_4.

We've connected 5 GPU's through riser cables:
https://eu.mouser.com/ProductDetail/517 ... -0726-0500

[Ron]

Thanks for sharing your experience and your solutions.
That is highly appreciated!

Ron

[mattking]

It's the connector nr. 2 on the Asus Manual Page 1_4.

We've connected 5 GPU's through riser cables:
https://eu.mouser.com/ProductDetail/517 ... -0726-0500

Thanks GIOLETS

No.2 in the manual is power connectors? I think my 1200W Corsair HX1200 Platinum power supply is connected to that connection, so I have a spare power connection for the GPU its more an issue of fitting the GPU with my Asus X99 E WS motherboard, I looked into use a PCI E riser, but I think I stopped looking into that as as option because if I use a riser, then there is no where to attach the GPU to my case. What did you guys do to attach the extra GPUs??

[alecmaassen]

Could you go more in depth about the TDR setting? I've never even heard of this before.