Render Failure CUDA error 702

[lakerwiz]

Hey Guys, first time posting here but a frequent user of the forums.

I'm having some Render Error problems since I've upgraded from Cinema 4D R17 to R18. I'm using the same scene as before, however now it will not render without giving me a Render Error and even crashing my computer. Upon clicking render, it takes a few minutes to finally show a first look at the scene in the Live Viewer, then it renders unbearably slow for about 6% before locking up my C4D and then PC. I've included all the necessary info below, as well as steps taken, additional steps of troubleshooting, versions, etc.

More info Below:

---

A. Windows 10 Pro 10.0.15063
B. 1X 980 ti Hybrid (6GB VRAM) + 2X 1080 Ti FE (11GB VRAM)
C. 64GB RAM
D. Nvidia driver 382.05
E. OctaneRender Standalone version 3.06
F. OctaneRender plugin version V3.06-win-test4 (current version, have same problems with V3.05.3 stable and V3.04.5).
G. Cinema 4D Studio R18.041
H. Description of Problem:

Upon rendering my scene, I received a bunch of error messages including some of the following: *please note* only including a few examples for immediate reference. Duplicate error messages appeared across all devices 0, 1, and 2. Full error log attached below for additional reference

-"This source doesn't contain any primitives!"
-"CUDA error 702 on device 1: the launch timed out and was terminated
-> failed to launch kernel(ptBrdf2)
device 1: direct light kernel failed
device 1: detected an error on render device! trying to recover...
CUDA error 702 on device 1: the launch timed out and was terminated
-> failed to deallocate device memory"
-"CUDA error 702 on device 1: the launch timed out and was terminated
-> could not get memory info
CUDA error 702 on device 1: the launch timed out and was terminated
-> failed to unload module"
-"CUDA error 702 on device 0: the launch timed out and was terminated
-> failed to bind device to current thread
device 0: failed to initialize resources
device 0: failed to recover - giving up and switching device to failed"
-"CUDA error 702 on device 2: the launch timed out and was terminated
-> failed to bind device to current thread
device 2: failed to initialize resources
device 2: failed to recover - giving up and switching device to failed
CUDA error 702 on device 1: the launch timed out and was terminated
-> failed to bind device to current thread
device 1: failed to initialize resources
device 1: failed to recover - giving up and switching device to failed
could not pick materials, because the geometry has changed and rendering is paused"

What I've tried:
-Removing all light sources and rendering without lights (read somewhere on the forums recently this could be an issue)
-Checking for non-octane materials, scene is clean.
-turning off different combinations of the GPUs, same result on any given one.
-Updating C4D (I'm currently on version 18.041 as for some reason it will not manually install the most recent 18.048 version).
-Trying different Octane C4D builds (was previously using V3.05.3-win, have tried build V3.06-win-test4 as I recall being mentioned to have fixed some recent CUDA issues and even went back to V3.04.5-win. None of these solutions worked, all presented the same problem).
-Checking fro Driver Updates (I'm currently on the most recent and up to date NVidia build 382.05).

What I'm going to try now as an alternate approach:
-Revert back to R17, my last attempt to fix as I have a tight deadline and need to get this rendered asap.
*EDIT: I have re-installed R17 and am having the same issues with V3.06-win-test4 now. Log attached below. Also no longer works with V3.05.3*


I. Octane Log Provided Below

---

Thanks for your assistance and reply in advance, it's much appreciated.

Sincerely,

Kyle

[aoktar]

Could you check it this build? viewtopic.php?f=85&t=60816
Also try to render with just one gpu of all and let me know what happened.

[Liketheriver]

Hi lakerwiz,

I have a list of thing to check, maybe it can help you:

1) subdivisions surfaces: check if render subdivisions are the same as viewport subdivisions, because this can create a lot of polys and reach your vram limit during the picture viewer rendering (live view render the geometry in the viewport, instead PV render the full geometry).

2) Large images: if you are rendering big images (4k for example) with several layer, this can fill the vram because octane store the image buffer in the gpu memory, try to disable render passes.

3) Vram: your max available vram is 6gb because of the 980ti, try to disable only the 980ti (device1) to use the full 11gb of the 1080 ti's
but keep in mind that 'cause of a problem in vram management in win10, a % (usually 20%) of the vram is not available for rendering, for example, in the 980ti you have only 4.4gb available.

4) Textures: if you are using large textures try to enable out of core textures in octane settings.

5) Geometry errors: If an object is corrupted or the geometry have some problem this can crash the render, to find out the object delete half the scene, then render, if it work the bad object is in the other half, undo and delete half of that part and go on until the render keep working and you end up with only one object (or more) to delete

good luck

[lakerwiz]

Could you check it this build? viewtopic.php?f=85&t=60816
Also try to render with just one gpu of all and let me know what happened.

Hey Aoktar,

Thanks for the reply. I went ahead and updated to the latest Octane C4D build V3.06 this morning and am still receiving the Render Error.

Here's a list of more solutions I've tried to resolve the issue:

-I have gone ahead and done further testing today, including the installation of the V3.06 version you linked. The problem still remains. I have V3.06 of the standalone version installer/ installed on my machine, and also tried using the standalone V3.06 zip version.
-Initially I had a hunch that Octane wasn’t liking the instances or a particular plugin (Forester) in my scene. I went ahead and removed all the assets using the plugin and removed all instancing so that just the object remain. Doing both had no effect and I still am experiencing the Render Error and crashing. I have also gone ahead and double checked all the textures and made sure they have the proper file locations, imagetexture tag, etc. None of these has fixed the issue after some further testing.
-I went ahead and tested as per your suggestion on individual cards, the scene still crashes even with different variations of 1 card enabled and the other 2 off.
-I did a clean install of my graphics drivers with the most up to date Nvidia driver, 382.05. The problem still remains.

Also not sure if it’s related, but in some cases the scene does render a frame in the Octane Viewer without any problems at all (false hope). However upon changing the camera angle or hitting render, it crashes. Haven’t been able to get it to render past a single frame. I also notice when it is rendering that even when amidst rendering, my GPUs will not remain in full use, and jump between 0% and 50-70%. I’ve attached further logs and screen shots as of the C4D Octane build downloaded this morning as well as what happens after the render crash occurs. I’ve also noticed it randomly will completely take one of my GPUs offline to the point where a PC restart is necessary unless it crashes my computer altogether.

Additionally, every few times it crashes I will get a blue screen windows crash with a DPC_WATCHDOG_VIOLATION error. I've gone ahead and done clean installs of all the associated drivers and system scans for issues with the registry, everything is now fresh and there are no system errors, however I'm still receiving the errors. The last thing I haven't done is reset the PC/ reinstalled Windows which could be a last resort measure.


Let me know if there's anything else you can think of that may be causing the problem. I will continue to tinker on my end. Thank you very much for your help and assistance in advance!

-Kyle

[lakerwiz]

Hi lakerwiz,

I have a list of thing to check, maybe it can help you:
)

Hey Liketheriver,

Thanks for taking the time to hop in. My replies below:

1) subdivisions surfaces: check if render subdivisions are the same as viewport subdivisions, because this can create a lot of polys and reach your vram limit during the picture viewer rendering (live view render the geometry in the viewport, instead PV render the full geometry).
-I'll look into this. Is there a specific checkbox in the C4D Plugin that enables this or modifies it?

2) Large images: if you are rendering big images (4k for example) with several layer, this can fill the vram because octane store the image buffer in the gpu memory, try to disable render passes.
-Rendering at 1080p. I've been successful in rendering this scene already completely with my previous build of C4D R17. Nothing has been changed in the scene, but now the client wants some minor changes to certain elements (making them black instead of textured, so this should actually save VRAM). Previously there were no issues with the VRAM maxing out, close but still no problems with my 3 GPUs enabled. The problem has just been occuring since I've updated to R18 and even though I have tried going back to R17, I am now having the problem consistently on both versions.

3) Vram: your max available vram is 6gb because of the 980ti, try to disable only the 980ti (device1) to use the full 11gb of the 1080 ti's
but keep in mind that 'cause of a problem in vram management in win10, a % (usually 20%) of the vram is not available for rendering, for example, in the 980ti you have only 4.4gb available.
-Noted, as I mentioned above, I didn't have this issue before, and when I did run into this issue in the past, it gave me different errors pointing towards maxed out VRAM and I optimized from there. It definitely didn't crash my whole PC.

4) Textures: if you are using large textures try to enable out of core textures in octane settings.
-I have this enabled by default. Any particular settings you recommend just in case?

5) Geometry errors: If an object is corrupted or the geometry have some problem this can crash the render, to find out the object delete half the scene, then render, if it work the bad object is in the other half, undo and delete half of that part and go on until the render keep working and you end up with only one object (or more) to delete
-Will give this a shot again. I did try this the other day but will maybe try each geometry one by one. There weren't any issues with the geometry as of my last successful render so I'm not sure what could have happened. Maybe I'll try loading up a previous version of the scene into R17 and see if maybe the upgrade to R18 corrupted the scene.

Thanks again for your assistance and pointers, appreciate the reply in advance.

-Kyle

[aoktar]

Your some gpus may have defect. Seeing corrupted pixels and even some bluescreen is giving this idea. Can you try with lowering the gpu speed/frequencies. Also another step is to remove some gpus and try again.

[lakerwiz]

Your some gpus may have defect. Seeing corrupted pixels and even some bluescreen is giving this idea. Can you try with lowering the gpu speed/frequencies. Also another step is to remove some gpus and try again.

Hey Aoktar,

Some more info/ updates for you:

-I have updated my BIOS and reset my CMOS. Problem persisted.
-I rolled back my Nvidia driver to the previous working version, problem persisted.
-*I copied all the scene data to a new scene and saved it differently. I am no longer receiving Render Errors! However, I don't believe the problem is solved as of yet as there is some wierd stuff going on with my 1080ti cards as you'll note by the graphs below. I also tried underclocking the cards and noted some difference in behavior by setting the card priorities all to "use priority"

I am now on version V3.06 stable for both the standalone and C4D Plugin, in Cinema 4D R18.


Let me know your thoughts based on the images below, thank you!

Sincerely,

Kyle

[proundkontra]

Hi
I have the same issues.
I tested all gpus seperately to make sure no gpu has a defect. Running with only one gpu everything works perfect as id should.
If i put all of them back together the problem is back there.
What nvidia driver do you use now?

[bepeg4d]

Hi proundkontra,
please check if the Nvidia control panel that the multiGPU/SLI option is disabled.
ciao beppe

[coilbook]

Your some gpus may have defect. Seeing corrupted pixels and even some bluescreen is giving this idea. Can you try with lowering the gpu speed/frequencies. Also another step is to remove some gpus and try again.

Hey Aoktar,

Some more info/ updates for you:

-I have updated my BIOS and reset my CMOS. Problem persisted.
-I rolled back my Nvidia driver to the previous working version, problem persisted.
-*I copied all the scene data to a new scene and saved it differently. I am no longer receiving Render Errors! However, I don't believe the problem is solved as of yet as there is some wierd stuff going on with my 1080ti cards as you'll note by the graphs below. I also tried underclocking the cards and noted some difference in behavior by setting the card priorities all to "use priority"

I am now on version V3.06 stable for both the standalone and C4D Plugin, in Cinema 4D R18.


Let me know your thoughts based on the images below, thank you!

Sincerely,

Kyle

hi were you able to fix it? I ma getting error with large scenes Thank you
CUDA error 719 on device 0: unspecified launch failure
-> kernel execution failed(report)
CUDA error 719 on device 0: unspecified launch failure
-> failed to launch kernel(ptBrdf2)
device 0: direct light kernel failed
CUDA error 719 on device 1: unspecified launch failure