CUDA 719 Issues !

[Lewis]

HI!

I'm having serious problems with rendering in build 3.03.3.0 (LW plugin)
It starts rendering and then after minute, two or 3 it just gives up, Windows screen freezes for 3-4 sec (sound works fine in BG so it's only GPU lockup). It's always different GPU (it's not hardware issue). Sometiem it says GPU given up or some such silly message.
I use latest 372.54 driver from nvidia, Windows 10 64bit AU (Anniversary Update) and Titanx+2* GTX 1080.

My windows registry TRDelay setings show this: value "0x0000000a (10)" . That owudl mena it's already set to 10 seconds by default ?

Please HELP, i need to finish this job and i cant' render anyhtign more than 4-5 minutes (t's not overheating or so, GPUs are 55-72 C). Thanks

Here is part of octane.log:

00:01:59 (0119.25) * OCTANE API MSG: CUDA error 719 on device 0: unspecified launch failure -> failed to bind device to current thread
00:01:59 (0119.25) * OCTANE API MSG: device 0: failed to initialize resources
00:01:59 (0119.25) * OCTANE API MSG: device 0: failed to recover - giving up and switching device to failed
00:01:59 (0119.25) |
00:01:59 (0119.25) | ++++++++++++++++++++++
00:01:59 (0119.25) | +++ RENDER FAILURE +++ processing the failure callback
00:01:59 (0119.25) | ++++++++++++++++++++++
00:01:59 (0119.25) |
00:01:59 (0119.25) |
00:01:59 (0119.25) | Aborting rendering detected
00:01:59 (0119.25) | >>> Refresh preview window
00:01:59 (0119.25) | >>> Draw preview window information
00:01:59 (0119.25) * OCTANE API MSG: CUDA error 719 on device 2: unspecified launch failure
00:01:59 (0119.25) * OCTANE API MSG: CUDA error 719 on device 2: unspecified launch failure -> failed to bind device to current thread
00:01:59 (0119.25) * OCTANE API MSG: device 2: failed to initialize resources
00:01:59 (0119.25) * OCTANE API MSG: device 2: failed to recover - giving up and switching device to failed
00:01:59 (0119.25) | >>> Draw preview window status
00:01:59 (0119.25) | >>> Draw preview window progress bar
00:01:59 (0119.25) | >>> Draw preview image
00:01:59 (0119.25) | >>> Draw preview image finished. Releaseing mutex
00:01:59 (0119.25) | >>> Refresh preview window done
00:01:59 (0119.25) | >>> Draw preview window information
00:01:59 (0119.25) | >>> Draw preview window status
00:01:59 (0119.25) | >>> Draw preview window progress bar
00:01:59 (0119.25) | >>> Draw preview image
00:01:59 (0119.26) | >>> Draw preview image finished. Releaseing mutex
00:01:59 (0119.26) | ... Finish Rendering ...
00:01:59 (0119.26) |
00:01:59 (0119.26) | ... wait for the getPreviewImage function ...
00:01:59 (0119.36) | ... reset image callback ...
00:01:59 (0119.36) | Close and free scene, free buffer: 1, reset scene: 1
00:01:59 (0119.36) | ... freeBuffers
00:01:59 (0119.37) | ... setRenderTargetNode(NULL)
00:01:59 (0119.47) | ... update()
00:01:59 (0119.47) | ... getRootNodeGraph()->clear()
00:01:59 (0119.52) * OCTANE API MSG: CUDA error 719 on device 1: unspecified launch failure
00:01:59 (0119.52) * OCTANE API MSG: CUDA error 719 on device 1: unspecified launch failure -> failed to bind device to current thread
00:01:59 (0119.52) * OCTANE API MSG: device 1: failed to initialize resources
00:01:59 (0119.52) * OCTANE API MSG: device 1: failed to recover - giving up and switching device to failed
00:01:59 (0119.77) | >>> Refresh preview window
00:01:59 (0119.77) | >>> Draw preview window progress bar
00:01:59 (0119.77) | >>> Refresh preview window done
00:01:59 (0119.77) | Scene closed
00:01:59 (0119.77) |
00:01:59 (0119.77) | Close preview window
00:01:59 (0119.77) |
00:01:59 (0119.77) | Octane Render for Lightwave, end of log system


Yesterday i was getting different messages but same result:

"00:04:19 (0259.61) * OCTANE API MSG: CUDA error 719 on device 2: unspecified launch failure -> failed to deallocate device memory
00:04:19 (0259.61) * OCTANE API MSG: CUDA error 719 on device 2: unspecified launch failure
00:04:19 (0259.61) * OCTANE API MSG: CUDA error 719 on device 2: unspecified launch failure -> could not get memory info
00:04:19 (0259.61) * OCTANE API MSG: CUDA error 719 on device 2: unspecified launch failure
00:04:19 (0259.61) * OCTANE API MSG: CUDA error 719 on device 2: unspecified launch failure -> failed to deallocate device memory
00:04:19 (0259.61) * OCTANE API MSG: CUDA error 719 on device 2: unspecified launch failure
00:04:19 (0259.61) * OCTANE API MSG: CUDA error 719 on device 2: unspecified launch failure -> could not get memory info
00:04:19 (0259.61) * OCTANE API MSG: CUDA error 719 on device 2: unspecified launch failure
00:04:19 (0259.61) * OCTANE API MSG: CUDA error 719 on device 2: unspecified launch failure -> failed to unload module
00:04:32 (0272.74) | IPR: abort render thread command finished
00:04:32 (0272.74) | Close and free scene, free buffer: 0, reset scene: 1
00:04:32 (0272.74) | ... setRenderTargetNode(NULL)
00:04:32 (0272.74) | ... update()
00:04:32 (0272.74) | ... getRootNodeGraph()->clear()
00:04:33 (0273.00) | Scene closed
00:04:33 (0273.00) |
00:04:33 (0273.00) | IPR: free the snapshots system
00:04:33 (0273.00) | IPR: IPR preview window closed
00:04:33 (0273.00) |
00:04:33 (0273.00) | Octane Render for Lightwave, end of log system"

[BorisGoreta]

I don't understand any of this but it seems you jumped to 1080 a bit too early.

[Lewis]

Not really Pascal issue 'coz it happens sometime with TitanX and 780 6GB OC also.

[Lewis]

Hmm, it seems some foliage instancing is tripping it over, when i remove 3D instanced grass (100k grass patches) then it renders. Then i added some more trees and now it's aborting/crashing again. Maybe Opacity/clipmaps is making problems for octane. Any of those models work fine alone in scene.

[Lewis]

check this Error from earlier today. That's what windows app center gave me as message after 5-6 sec freeze while Octane was rendering and then aborting render. After that i had to close layout, photoshop and DWG viewer, they all "lost" openGL/image in programs.

[stef]

Hi !
Have you disable TDR ?
Or set a bigger delay timing ?

http://http.developer.nvidia.com/Nsight ... covery.htm
https://msdn.microsoft.com/en-us/librar ... s.85).aspx
I don't found these variables for W10, are they the same ?

Reboot is must need for process !
And cheap primary card to !
Best regards.
_

[Lewis]

Hi !
Have you disable TDR ?
Or set a bigger delay timing ?

http://http.developer.nvidia.com/Nsight ... covery.htm
https://msdn.microsoft.com/en-us/librar ... s.85).aspx
I don't found these variables for W10, are they the same ?

Reboot is must need for process !
And cheap primary card to !
Best regards.
_

I have it set for 10 seconds on my windows. What's your settings.

Thanks

[stef]

Hi,
I disable TDR !

Others things may help, or not .. For rendering,
- Switch LW in bounding box ( cpu render )
- Update bios firmware, and PCI drivers
- PC power options to the max perf
- Optimise all instanced objects
- Good air flow in the office
- Leave monitor ON ..
- Last GTX drivers

Good luck !
_

[Lewis]

Hi,
I disable TDR !
_

Can you post screen-grab/settings of your registry how to disable it ? Thanks

BTW this 719 is definitely Octane BUG/problem. By now I've tested this scene on 3 machines so far, different GPUs (Maxwel/Pascal/Kepler all in mix and all in separate) and i've even gone to octane 2.25 and they both "die" after some time of rendering with that CUDA 719 log/problem. I've even gone so far that i packed scene (1GB in total RAR) and sent it to another user (Thanks Oliver) and he run it on his Machine and it died after 20-30 minutes also.

So far tested on 3 PCs, 3 windows versions (win7, win8, win10) and 10 GPUs in total tested (TitanX, 2*780, 2*1080, 1*780Ti, 4*980Ti) and it always dies with that silly 719 LOG error message. Sometime after 2 minutes sometime even after 2 hours of rendering but it always fail.

Must be Octane BUG then.

P.S. I've even re-modeled geometry (it was crappy Sketchup export model full of N-gons and Non planars...) form scratch to make it QuadsTris only and no N-gons and tripled all non-planars and still same issue. I tried PT and PMC kernels, no difference.

[LightwaveGuru]

https://www.txcorp.com/images/docs/gpul ... codes.html


cudaErrorInvalidTextureBinding = 19, ///< Invalid texture binding

lwguru