Render failed

Forums: Render failed
Sub forum for bug reports
Forum rules
Before posting a bug report, please check the following:
1. That the issue has not already been disclosed
2. That the issue is specific to this plugin, and not Octane in general (Try reproducing it in Standalone)
Bugs related to the Octane Engine itself should be posted into the Standalone Support sub-forum.


All bug reports should include the information below, along with a detailed description of the issue and steps to reproduce it.
A. Operating System, including version (i.e. Win 7, OSX 10.11.2, Ubuntu 14.04, etc.)
B. Graphics Card(s) model (i.e. GTX 580 - 3GB, TITAN, etc.)
C. RAM Capacity (i.e. 6 GB)
D. Nvidia driver version (i.e. 7.50, 7.5.22)
E. OctaneRender Standalone version, if installed (i.e. 2.24.2, 2.23, etc.)
F. OctaneRender plugin version (i.e. v2.25 - 2.21)
G. Host application version, including build number if available (i.e. 3ds Max 2016 Build 18.0)

Render failed

Postby coilbook » Sat Aug 05, 2017 9:01 pm

coilbook Sat Aug 05, 2017 9:01 pm
For some reason opening octane viewport with some VDBs gives me render failed message and then it will never open again. Max must be restarted

CUDA error 719 on device 0: unspecified launch failure
-> kernel execution failed(report)
CUDA error 719 on device 0: unspecified launch failure
-> failed to launch kernel(ptBrdf2)
device 0: direct light kernel failed
CUDA error 719 on device 1: unspecified launch failure
-> kernel execution failed(report)
CUDA error 719 on device 1: unspecified launch failure
-> failed to launch kernel(ptBrdf2)
device 1: direct light kernel failed
CUDA error 719 on device 3: unspecified launch failure
-> kernel execution failed(report)
CUDA error 719 on device 3: unspecified launch failure
-> failed to launch kernel(ptBrdf2)
device 3: direct light kernel failed
CUDA error 719 on device 0: unspecified launch failure
-> could not get memory info
CUDA error 719 on device 0: unspecified launch failure
-> could not get memory info
CUDA error 719 on device 1: unspecified launch failure
-> could not get memory info
CUDA error 719 on device 1: unspecified launch failure
-> could not get memory info
CUDA error 719 on device 3: unspecified launch failure
-> failed to unregister pinned memory
CUDA error 719 on device 3: unspecified launch failure
-> could not get memory info
CUDA error 719 on device 3: unspecified launch failure
-> failed to unregister pinned memory
CUDA error 719 on device 3: unspecified launch failure
-> could not get memory info
CUDA error 719 on device 1: unspecified launch failure
-> failed to load symbol data to the device(deep_data)
CUDA error 719 on device 0: unspecified launch failure
CUDA error 719 on device 3: unspecified launch failure
-> failed to load symbol data to the device(deep_data)
device 1: failed to upload the deep params
-> failed to load symbol data to the device(deep_data)
device 0: failed to upload the deep params
device 3: failed to upload the deep params
CUDA error 719 on device 0: unspecified launch failure
-> failed to deallocate device memory
CUDA error 719 on device 0: unspecified launch failure
-> could not get memory info
CUDA error 719 on device 0: unspecified launch failure
-> failed to deallocate device memory
CUDA error 719 on device 0: unspecified launch failure
-> could not get memory info
CUDA error 719 on device 0: unspecified launch failure
-> failed to deallocate device memory
CUDA error 719 on device 0: unspecified launch failure
-> could not get memory info
CUDA error 719 on device 0: unspecified launch failure
-> failed to deallocate device memory
CUDA error 719 on device 0: unspecified launch failure
-> could not get memory info
CUDA error 719 on device 0: unspecified launch failure
-> failed to deallocate device memory
CUDA error 719 on device 0: unspecified launch failure
-> could not get memory info
CUDA error 719 on device 0: unspecified launch failure
-> failed to deallocate device memory
CUDA error 719 on device 0: unspecified launch failure
-> could not get memory info
CUDA error 719 on device 0: unspecified launch failure
-> failed to deallocate device memory
CUDA error 719 on device 0: unspecified launch failure
-> could not get memory info
CUDA error 719 on device 0: unspecified launch failure
-> failed to deallocate device memory
CUDA error 719 on device 0: unspecified launch failure
-> could not get memory info
CUDA error 719 on device 0: unspecified launch failure
-> failed to deallocate device memory
CUDA error 719 on device 0: unspecified launch failure
-> could not get memory info
CUDA error 719 on device 0: unspecified launch failure
-> failed to deallocate device memory
CUDA error 719 on device 0: unspecified launch failure
-> could not get memory info
CUDA error 719 on device 0: unspecified launch failure
-> failed to deallocate device memory
CUDA error 719 on device 0: unspecified launch failure
-> could not get memory info
CUDA error 719 on device 0: unspecified launch failure
-> failed to deallocate device memory
CUDA error 719 on device 0: unspecified launch failure
-> could not get memory info
CUDA error 719 on device 0: unspecified launch failure
-> failed to deallocate device memory
CUDA error 719 on device 0: unspecified launch failure
-> could not get memory info
CUDA error 719 on device 0: unspecified launch failure
-> failed to deallocate device memory
CUDA error 719 on device 0: unspecified launch failure
-> could not get memory info
CUDA error 719 on device 0: unspecified launch failure
-> failed to deallocate device memory
CUDA error 719 on device 0: unspecified launch failure
-> could not get memory info
CUDA error 719 on device 0: unspecified launch failure
-> failed to deallocate device memory
CUDA error 719 on device 0: unspecified launch failure
-> could not get memory info
coilbook
Licensed Customer
Licensed Customer
 
Posts: 2612
Joined: Mon Mar 24, 2014 2:27 pm

Re: Render failed

Postby coilbook » Sun Aug 06, 2017 4:16 pm

coilbook Sun Aug 06, 2017 4:16 pm
Hi Otoy,
when slave gets error do I see in the master error log window or only on slave PC

and what does it mean CUDA error 719 unspecified launch failure? failed to upload the deep params?

it would render 10 frames and give this erro we are using itoo forest with lots of grass that is also generated when camera moves

Also we have this train pass hauling rocks as displacement map? Can this be the reason Or are we running out of vram

Also after crash 3ds max must be restarted or rendering screen wil ljsut stay black
Attachments
cuda error.jpg
coilbook
Licensed Customer
Licensed Customer
 
Posts: 2612
Joined: Mon Mar 24, 2014 2:27 pm

Re: Render failed

Postby coilbook » Sun Aug 06, 2017 4:30 pm

coilbook Sun Aug 06, 2017 4:30 pm
so this bug is similar to this viewtopic.php?f=36&t=55994&start=10

hopefully it can be fixed

"esterday, I finally found and fixed a long standing issue with the displacement intersection test, which sometimes can end up in an infinite loop, causing a kernel to never finish. The OS will then eventually pull the plug as you have reported above. The fix will be available with the next release. Obviously, this will only help in scene with displacement mapping. If you experience similar issues in scenes without displacement mapping, please send them to either Juanjo or to us so we can investigate. Thank you."
coilbook
Licensed Customer
Licensed Customer
 
Posts: 2612
Joined: Mon Mar 24, 2014 2:27 pm

Re: Render failed

Postby coilbook » Sun Aug 06, 2017 5:32 pm

coilbook Sun Aug 06, 2017 5:32 pm
ok after some tests it looks like one of the slave crashes with the following error as well making rendering stop

cuda 999 could not get memory info and one of the GPus disappears from evga precision. But that GPU works greate rendering other scenes so it is not bad GPU I guess .

Another slave just gave me an error Cuda context was not destroyed
Seems like all these errors above happen after 10 frames of rendering. Also railroad rocks is a displacement map


RENDER TIMES:
20 GPUs - 2:09 min

14GPUs - 2:20 is network a problem? It is crazy 4 1080s and 2 Titan X pascal only give me 11 seconds of render time speed?
Attachments
IMG_3208.JPG
coilbook
Licensed Customer
Licensed Customer
 
Posts: 2612
Joined: Mon Mar 24, 2014 2:27 pm

Re: Render failed

Postby coilbook » Sun Aug 06, 2017 9:50 pm

coilbook Sun Aug 06, 2017 9:50 pm
Here are more errors on this slave. Not sure why this slave. SOme slaves are fine. It seems like octane cannot handle vast spaces full of trees and grass especially pathtracing we had similar problems before for opening shots and use of volume medium
Attachments
IMG_3210.JPG
IMG_3211.JPG
coilbook
Licensed Customer
Licensed Customer
 
Posts: 2612
Joined: Mon Mar 24, 2014 2:27 pm

Re: Render failed

Postby paride4331 » Mon Aug 07, 2017 6:36 am

paride4331 Mon Aug 07, 2017 6:36 am
Hi Coilbook,
Another user had a very similar issue to yours a few days ago, Cuda error 719 on two PCs in a 10 PC network.
However, it also happens without displacement map, it always happens.
Does your issue only happen with displacement map and usign cpu-memory?
All PMC. PT, DL-D,DL-AO?
Personally and I hope not to say wrong, but Cuda 7xx are problems with GPUs or drivers,
generally, and not directly related to Octane.
Regards
Paride
2 x Evga Titan X Hybrid / 3 x Evga RTX 2070 super Hybrid
User avatar
paride4331
Octane Guru
Octane Guru
 
Posts: 3011
Joined: Fri Sep 18, 2015 7:19 am

Re: Render failed

Postby coilbook » Mon Aug 07, 2017 1:48 pm

coilbook Mon Aug 07, 2017 1:48 pm
paride4331 wrote:Hi Coilbook,
Another user had a very similar issue to yours a few days ago, Cuda error 719 on two PCs in a 10 PC network.
However, it also happens without displacement map, it always happens.
Does your issue only happen with displacement map and usign cpu-memory?
All PMC. PT, DL-D,DL-AO?
Personally and I hope not to say wrong, but Cuda 7xx are problems with GPUs or drivers,
generally, and not directly related to Octane.
Regards
Paride


HI Paride,
after many tests we changed Kernel type from path tracing to Difuse AO and it rendered all night no problem. So there is a problem with path tracing engine or video cards cannot handle lots of objects with pathtracing. So displacement has nothing to do with it but maybe scene complexity and pathtracing. Also 999 and 719 errors were coming up on all slaves and master. I will try to send the scene at least without texture but it is over 1 GB in size
coilbook
Licensed Customer
Licensed Customer
 
Posts: 2612
Joined: Mon Mar 24, 2014 2:27 pm

Re: Render failed

Postby paride4331 » Mon Aug 07, 2017 2:49 pm

paride4331 Mon Aug 07, 2017 2:49 pm
Hi coilbook,
Just out of curiosity, in your network, How much Vram your video card with less Vram has?
Are you usign that with windows 10 or 7?
Regards
Paride
2 x Evga Titan X Hybrid / 3 x Evga RTX 2070 super Hybrid
User avatar
paride4331
Octane Guru
Octane Guru
 
Posts: 3011
Joined: Fri Sep 18, 2015 7:19 am

Re: Render failed

Postby coilbook » Mon Aug 07, 2017 3:37 pm

coilbook Mon Aug 07, 2017 3:37 pm
paride4331 wrote:Hi coilbook,
Just out of curiosity, in your network, How much Vram your video card with less Vram has?
Are you usign that with windows 10 or 7?
Regards
Paride



We have 1080s as lowest vram card. So with windows 10 we have 6.5Gb available the scene was 5.7 Gb Also each slave has extra 2Gb of RAM I guess if you run out of vram octane will use 2 gb of ram? Not sure if pathtracing uses more vram while rendering. Crash happened after 10-20 frames of rendering

we have 1 slave with windows 10, 3 with windows 7 and master windows 10

also those problems never happened before it happens more when camera is flying over a large opening scene maybe too much grass and trees to render?
coilbook
Licensed Customer
Licensed Customer
 
Posts: 2612
Joined: Mon Mar 24, 2014 2:27 pm

Re: Render failed

Postby coilbook » Mon Aug 07, 2017 7:14 pm

coilbook Mon Aug 07, 2017 7:14 pm
Hi Paride,
I sent you 2 private messages with the scene link to see why pathtracing makes it crash, slow octane viewport opening and fog density that goes from too much to nothing

UPDATE:

Can you also test rendering camera 4 frames 1200-1300. We have 20 GPUs render time is 2:30. 2 min is just a black screen and 30 seconds is to render. I thought most data is stored in vram already why it takes 2 min before each frame rendered to evaluate



Thank you
coilbook
Licensed Customer
Licensed Customer
 
Posts: 2612
Joined: Mon Mar 24, 2014 2:27 pm
Next

Return to Bug Reports


Who is online

Users browsing this forum: No registered users and 2 guests

Thu Jul 02, 2020 10:12 am [ UTC ]