OTOY Forums

For some reason opening octane viewport with some VDBs gives me render failed message and then it will never open again. Max must be restarted

CUDA error 719 on device 0: unspecified launch failure
-> kernel execution failed(report)
CUDA error 719 on device 0: unspecified launch failure
-> failed to launch kernel(ptBrdf2)
device 0: direct light kernel failed
CUDA error 719 on device 1: unspecified launch failure
-> kernel execution failed(report)
CUDA error 719 on device 1: unspecified launch failure
-> failed to launch kernel(ptBrdf2)
device 1: direct light kernel failed
CUDA error 719 on device 3: unspecified launch failure
-> kernel execution failed(report)
CUDA error 719 on device 3: unspecified launch failure
-> failed to launch kernel(ptBrdf2)
device 3: direct light kernel failed
CUDA error 719 on device 0: unspecified launch failure
-> could not get memory info
CUDA error 719 on device 0: unspecified launch failure
-> could not get memory info
CUDA error 719 on device 1: unspecified launch failure
-> could not get memory info
CUDA error 719 on device 1: unspecified launch failure
-> could not get memory info
CUDA error 719 on device 3: unspecified launch failure
-> failed to unregister pinned memory
CUDA error 719 on device 3: unspecified launch failure
-> could not get memory info
CUDA error 719 on device 3: unspecified launch failure
-> failed to unregister pinned memory
CUDA error 719 on device 3: unspecified launch failure
-> could not get memory info
CUDA error 719 on device 1: unspecified launch failure
-> failed to load symbol data to the device(deep_data)
CUDA error 719 on device 0: unspecified launch failure
CUDA error 719 on device 3: unspecified launch failure
-> failed to load symbol data to the device(deep_data)
device 1: failed to upload the deep params
-> failed to load symbol data to the device(deep_data)
device 0: failed to upload the deep params
device 3: failed to upload the deep params
CUDA error 719 on device 0: unspecified launch failure
-> failed to deallocate device memory
CUDA error 719 on device 0: unspecified launch failure
-> could not get memory info
CUDA error 719 on device 0: unspecified launch failure
-> failed to deallocate device memory
CUDA error 719 on device 0: unspecified launch failure
-> could not get memory info
CUDA error 719 on device 0: unspecified launch failure
-> failed to deallocate device memory
CUDA error 719 on device 0: unspecified launch failure
-> could not get memory info
CUDA error 719 on device 0: unspecified launch failure
-> failed to deallocate device memory
CUDA error 719 on device 0: unspecified launch failure
-> could not get memory info
CUDA error 719 on device 0: unspecified launch failure
-> failed to deallocate device memory
CUDA error 719 on device 0: unspecified launch failure
-> could not get memory info
CUDA error 719 on device 0: unspecified launch failure
-> failed to deallocate device memory
CUDA error 719 on device 0: unspecified launch failure
-> could not get memory info
CUDA error 719 on device 0: unspecified launch failure
-> failed to deallocate device memory
CUDA error 719 on device 0: unspecified launch failure
-> could not get memory info
CUDA error 719 on device 0: unspecified launch failure
-> failed to deallocate device memory
CUDA error 719 on device 0: unspecified launch failure
-> could not get memory info
CUDA error 719 on device 0: unspecified launch failure
-> failed to deallocate device memory
CUDA error 719 on device 0: unspecified launch failure
-> could not get memory info
CUDA error 719 on device 0: unspecified launch failure
-> failed to deallocate device memory
CUDA error 719 on device 0: unspecified launch failure
-> could not get memory info
CUDA error 719 on device 0: unspecified launch failure
-> failed to deallocate device memory
CUDA error 719 on device 0: unspecified launch failure
-> could not get memory info
CUDA error 719 on device 0: unspecified launch failure
-> failed to deallocate device memory
CUDA error 719 on device 0: unspecified launch failure
-> could not get memory info
CUDA error 719 on device 0: unspecified launch failure
-> failed to deallocate device memory
CUDA error 719 on device 0: unspecified launch failure
-> could not get memory info
CUDA error 719 on device 0: unspecified launch failure
-> failed to deallocate device memory
CUDA error 719 on device 0: unspecified launch failure
-> could not get memory info
CUDA error 719 on device 0: unspecified launch failure
-> failed to deallocate device memory
CUDA error 719 on device 0: unspecified launch failure
-> could not get memory info
CUDA error 719 on device 0: unspecified launch failure
-> failed to deallocate device memory
CUDA error 719 on device 0: unspecified launch failure
-> could not get memory info

Hi Otoy,
when slave gets error do I see in the master error log window or only on slave PC

and what does it mean CUDA error 719 unspecified launch failure? failed to upload the deep params?

it would render 10 frames and give this erro we are using itoo forest with lots of grass that is also generated when camera moves

Also we have this train pass hauling rocks as displacement map? Can this be the reason Or are we running out of vram

Also after crash 3ds max must be restarted or rendering screen wil ljsut stay black

so this bug is similar to this viewtopic.php?f=36&t=55994&start=10

hopefully it can be fixed

"esterday, I finally found and fixed a long standing issue with the displacement intersection test, which sometimes can end up in an infinite loop, causing a kernel to never finish. The OS will then eventually pull the plug as you have reported above. The fix will be available with the next release. Obviously, this will only help in scene with displacement mapping. If you experience similar issues in scenes without displacement mapping, please send them to either Juanjo or to us so we can investigate. Thank you."

ok after some tests it looks like one of the slave crashes with the following error as well making rendering stop

cuda 999 could not get memory info and one of the GPus disappears from evga precision. But that GPU works greate rendering other scenes so it is not bad GPU I guess .

Another slave just gave me an error Cuda context was not destroyed
Seems like all these errors above happen after 10 frames of rendering. Also railroad rocks is a displacement map


RENDER TIMES:
20 GPUs - 2:09 min

14GPUs - 2:20 is network a problem? It is crazy 4 1080s and 2 Titan X pascal only give me 11 seconds of render time speed?

Here are more errors on this slave. Not sure why this slave. SOme slaves are fine. It seems like octane cannot handle vast spaces full of trees and grass especially pathtracing we had similar problems before for opening shots and use of volume medium

Hi Coilbook,
Another user had a very similar issue to yours a few days ago, Cuda error 719 on two PCs in a 10 PC network.
However, it also happens without displacement map, it always happens.
Does your issue only happen with displacement map and usign cpu-memory?
All PMC. PT, DL-D,DL-AO?
Personally and I hope not to say wrong, but Cuda 7xx are problems with GPUs or drivers,
generally, and not directly related to Octane.
Regards
Paride

paride4331 wrote:Hi Coilbook,
Another user had a very similar issue to yours a few days ago, Cuda error 719 on two PCs in a 10 PC network.
However, it also happens without displacement map, it always happens.
Does your issue only happen with displacement map and usign cpu-memory?
All PMC. PT, DL-D,DL-AO?
Personally and I hope not to say wrong, but Cuda 7xx are problems with GPUs or drivers,
generally, and not directly related to Octane.
Regards
Paride

HI Paride,
after many tests we changed Kernel type from path tracing to Difuse AO and it rendered all night no problem. So there is a problem with path tracing engine or video cards cannot handle lots of objects with pathtracing. So displacement has nothing to do with it but maybe scene complexity and pathtracing. Also 999 and 719 errors were coming up on all slaves and master. I will try to send the scene at least without texture but it is over 1 GB in size

Hi coilbook,
Just out of curiosity, in your network, How much Vram your video card with less Vram has?
Are you usign that with windows 10 or 7?
Regards
Paride

paride4331 wrote:Hi coilbook,
Just out of curiosity, in your network, How much Vram your video card with less Vram has?
Are you usign that with windows 10 or 7?
Regards
Paride

We have 1080s as lowest vram card. So with windows 10 we have 6.5Gb available the scene was 5.7 Gb Also each slave has extra 2Gb of RAM I guess if you run out of vram octane will use 2 gb of ram? Not sure if pathtracing uses more vram while rendering. Crash happened after 10-20 frames of rendering

we have 1 slave with windows 10, 3 with windows 7 and master windows 10

also those problems never happened before it happens more when camera is flying over a large opening scene maybe too much grass and trees to render?

Hi Paride,
I sent you 2 private messages with the scene link to see why pathtracing makes it crash, slow octane viewport opening and fog density that goes from too much to nothing

UPDATE:

Can you also test rendering camera 4 frames 1200-1300. We have 20 GPUs render time is 2:30. 2 min is just a black screen and 30 seconds is to render. I thought most data is stored in vram already why it takes 2 min before each frame rendered to evaluate



Thank you