Batch render from command line crash

[sampakinpa]

Number of UV indices: 752
Number of material indices: 161
0:02 ...done setting Mesh data in 0:00
0:02 Voxelizing...
0:02 ...voxelized in 0:00

0:02 Setting Mesh data of mesh: Ring_8825RND:D_White_5Shape
****************************** MESH DATA ******************************
Number of vertices: 217
Number of normals: 752
Number of UVs: 657
Number of polygons: 161
Number of vertex indices: 752
Number of normal indices: 752
Number of UV indices: 752
Number of material indices: 161
0:02 ...done setting Mesh data in 0:00
0:02 Voxelizing...
0:02 ...voxelized in 0:00

0:02 Setting Mesh data of mesh: Ring_8825RND:D_White_6Shape
****************************** MESH DATA ******************************
Number of vertices: 217
Number of normals: 752
Number of UVs: 657
Number of polygons: 161
Number of vertex indices: 752
Number of normal indices: 752
Number of UV indices: 752
Number of material indices: 161
0:02 ...done setting Mesh data in 0:00
0:02 Voxelizing...
0:02 ...voxelized in 0:00

0:02 Setting Mesh data of mesh: Ring_8825RND:G_White_1Shape
****************************** MESH DATA ******************************
Number of vertices: 10775
Number of normals: 10775
Number of UVs: 11538
Number of polygons: 10688
Number of vertex indices: 42752
Number of normal indices: 42752
Number of UV indices: 42752
Number of material indices: 10688
0:02 ...done setting Mesh data in 0:00
0:02 Voxelizing...
0:02 ...voxelized in 0:00

0:02 Setting Mesh data of mesh: Ring_8825RND:G_White_3Shape
****************************** MESH DATA ******************************
Number of vertices: 12800
Number of normals: 12800
Number of UVs: 12800
Number of polygons: 12288
Number of vertex indices: 49152
Number of normal indices: 49152
Number of UV indices: 49152
Number of material indices: 12288
0:02 ...done setting Mesh data in 0:00
0:02 Voxelizing...
0:02 ...voxelized in 0:00

0:02 Setting Mesh data of mesh: Ring_8825RND:Prong_2Shape
****************************** MESH DATA ******************************
Number of vertices: 18650
Number of normals: 18830
Number of UVs: 19620
Number of polygons: 18688
Number of vertex indices: 74752
Number of normal indices: 74752
Number of UV indices: 74752
Number of material indices: 18688
0:02 ...done setting Mesh data in 0:00
0:02 Voxelizing...
0:02 ...voxelized in 0:00

0:02 Setting Mesh data of mesh: Ring_8825RND:Soliter_1Shape
****************************** MESH DATA ******************************
Number of vertices: 217
Number of normals: 752
Number of UVs: 657
Number of polygons: 161
Number of vertex indices: 752
Number of normal indices: 752
Number of UV indices: 752
Number of material indices: 161
0:02 ...done setting Mesh data in 0:00
0:02 Voxelizing...
0:02 ...voxelized in 0:00
0:02 ...done,
0:02 Gathering Subdiv objects data...
0:02 ...done,
0:02 Gathering Octane Light objects data...
0:02 ...done,
0:02 Gathering Instancer objects data...
0:02 ...done,

0:02 Setting Mesh data of general mesh
****************************** GENERAL MESH DATA ******************************
Number of vertices: 43758
Number of normals: 47693
Number of UVs: 48573
Number of polygons: 42797
Number of material indices: 42797
Number of vertex indices: 171944
Number of normal indices: 171944
Number of UV indices: 171944
0:02 ...done setting Mesh data in 0:00
0:02 Voxelizing...
0:02 ...voxelized in 0:00
0:02 ...done creating meshes
0:02 Octane mesh data is set to render engine.
0:02 Octane kernel set: Path Tracing

****************************** THIN LENS CAMERA ********************************
*
Name: cameraShape1
Orthographic: 0
Film width: 35.999928, Focal length: 45, calculated FOV: 43.602741
Aperture: 0.03
Auto-focus: 1
Focal depth: 2.381014
Stereo: 0
Stereo distance: 0.001
Near clip depth: 1e-007
Camera: X: -1.894747, Y: 3.952579, Z: 0.290888
Target: X: 0.773788, Y: 1.149091, Z: -0.1129
Up: X: 0.712307, Y: 0.693543, Z: -0.107783
IMAGER:
Exposure: 0.8
FStop: 2.8
Iso: 100
Gamma: 0.7
Response: 400
Vignetting: 0.3
Saturation: 1
Hot pixel removal: 1
Premultiplied alpha: 0
Min display samples: 1
Dithering: 0
White saturation: 1
POSTPROCESSOR:
Enabled: 1
Bloom power: 20
Glare power: 15
Glare ray count: 4
Glare angle: 31.263159
Glare blur: 0.0491789

****************************** TEXTURE ENVIRONMENT ********************
Sun/Sky Node: octaneSunSky
Power: 1.8
Rotation: X 0.6, Y -0.3

0:04 Octane engine: Target node is set...
0:04 Octane: Scene export complete.
0:00 ...done
Octane: Refreshed image, 38 samples per pixel of 16000
Octane: Refreshed image, 113 samples per pixel of 16000
Octane: Refreshed image, 171 samples per pixel of 16000
Octane: Refreshed image, 258 samples per pixel of 16000
Octane: Refreshed image, 337 samples per pixel of 16000
Octane: Refreshed image, 384 samples per pixel of 16000
Octane: Refreshed image, 497 samples per pixel of 16000
Octane: Refreshed image, 562 samples per pixel of 16000
Octane: Refreshed image, 636 samples per pixel of 16000
Octane: Refreshed image, 720 samples per pixel of 16000
Octane: Refreshed image, 816 samples per pixel of 16000
Octane: Refreshed image, 925 samples per pixel of 16000
Octane: Refreshed image, 1047 samples per pixel of 16000
Octane: Refreshed image, 1180 samples per pixel of 16000
Octane: Refreshed image, 1313 samples per pixel of 16000
Octane: Refreshed image, 1445 samples per pixel of 16000
Octane: Refreshed image, 1577 samples per pixel of 16000
Octane: Refreshed image, 1709 samples per pixel of 16000
Octane: Refreshed image, 1841 samples per pixel of 16000
Octane: Refreshed image, 1972 samples per pixel of 16000
Octane: Refreshed image, 2105 samples per pixel of 16000
Octane: Refreshed image, 2237 samples per pixel of 16000
Octane: Refreshed image, 2368 samples per pixel of 16000
Octane: Refreshed image, 2498 samples per pixel of 16000
Octane: Refreshed image, 2629 samples per pixel of 16000
Octane: Refreshed image, 2763 samples per pixel of 16000
Octane: Refreshed image, 2897 samples per pixel of 16000
Octane: Refreshed image, 3029 samples per pixel of 16000
Octane: Refreshed image, 3162 samples per pixel of 16000
Octane: Refreshed image, 3294 samples per pixel of 16000
Octane: Refreshed image, 3425 samples per pixel of 16000
Octane: Refreshed image, 3557 samples per pixel of 16000
Octane: Refreshed image, 3689 samples per pixel of 16000
Octane: Refreshed image, 3820 samples per pixel of 16000
Octane: Refreshed image, 3951 samples per pixel of 16000
Octane: Refreshed image, 4081 samples per pixel of 16000
Octane: Refreshed image, 4213 samples per pixel of 16000
Octane: Refreshed image, 4345 samples per pixel of 16000
Octane: Refreshed image, 4476 samples per pixel of 16000
Octane: Refreshed image, 4607 samples per pixel of 16000
Octane: Refreshed image, 4738 samples per pixel of 16000
Octane: Refreshed image, 4869 samples per pixel of 16000
Octane: Refreshed image, 5002 samples per pixel of 16000
Octane: Refreshed image, 5134 samples per pixel of 16000
Octane: Refreshed image, 5267 samples per pixel of 16000
Octane: Refreshed image, 5399 samples per pixel of 16000
Octane: Refreshed image, 5530 samples per pixel of 16000
Octane: Refreshed image, 5661 samples per pixel of 16000
Octane: Refreshed image, 5791 samples per pixel of 16000
Octane: Refreshed image, 5923 samples per pixel of 16000
Octane: Refreshed image, 6055 samples per pixel of 16000
Octane: Refreshed image, 6186 samples per pixel of 16000
Octane: Refreshed image, 6317 samples per pixel of 16000
Octane: Refreshed image, 6448 samples per pixel of 16000
Octane: Refreshed image, 6580 samples per pixel of 16000
Octane: Refreshed image, 6711 samples per pixel of 16000
Octane: Refreshed image, 6842 samples per pixel of 16000
Octane: Refreshed image, 6975 samples per pixel of 16000
Octane: Refreshed image, 7107 samples per pixel of 16000
Octane: Refreshed image, 7240 samples per pixel of 16000
Octane: Refreshed image, 7373 samples per pixel of 16000
Octane: Refreshed image, 7504 samples per pixel of 16000
Octane: Refreshed image, 7637 samples per pixel of 16000
Octane: Refreshed image, 7770 samples per pixel of 16000
Octane: Refreshed image, 7903 samples per pixel of 16000
Octane: Refreshed image, 8035 samples per pixel of 16000
Octane: Refreshed image, 8167 samples per pixel of 16000
Octane: Refreshed image, 8299 samples per pixel of 16000
Octane: Refreshed image, 8432 samples per pixel of 16000
Octane: Refreshed image, 8564 samples per pixel of 16000
Octane: Refreshed image, 8699 samples per pixel of 16000
Octane: Refreshed image, 8830 samples per pixel of 16000
Octane: Refreshed image, 8962 samples per pixel of 16000
Octane: Refreshed image, 9095 samples per pixel of 16000
Octane: Refreshed image, 9228 samples per pixel of 16000
Octane: Refreshed image, 9361 samples per pixel of 16000
Octane: Refreshed image, 9493 samples per pixel of 16000
Octane: Refreshed image, 9627 samples per pixel of 16000
Octane: Refreshed image, 9761 samples per pixel of 16000
Octane: Refreshed image, 9894 samples per pixel of 16000
Octane: Refreshed image, 10027 samples per pixel of 16000
Octane: Refreshed image, 10159 samples per pixel of 16000
Octane: Refreshed image, 10291 samples per pixel of 16000
Octane: Refreshed image, 10424 samples per pixel of 16000
Octane: Refreshed image, 10556 samples per pixel of 16000
Octane: Refreshed image, 10689 samples per pixel of 16000
Octane: Refreshed image, 10821 samples per pixel of 16000
Octane: Refreshed image, 10953 samples per pixel of 16000
Octane: Refreshed image, 11085 samples per pixel of 16000
Octane: Refreshed image, 11216 samples per pixel of 16000
Octane: Refreshed image, 11348 samples per pixel of 16000
Octane: Refreshed image, 11481 samples per pixel of 16000
Octane: Refreshed image, 11613 samples per pixel of 16000
Octane: Refreshed image, 11745 samples per pixel of 16000
Octane: Refreshed image, 11878 samples per pixel of 16000
Octane: Refreshed image, 12009 samples per pixel of 16000
Octane: Refreshed image, 12140 samples per pixel of 16000
Octane: Refreshed image, 12273 samples per pixel of 16000
Octane: Refreshed image, 12405 samples per pixel of 16000
Octane: Refreshed image, 12537 samples per pixel of 16000
Octane: Refreshed image, 12668 samples per pixel of 16000
Octane: Refreshed image, 12799 samples per pixel of 16000
Octane: Refreshed image, 12929 samples per pixel of 16000
Octane: Refreshed image, 13061 samples per pixel of 16000
Octane: Refreshed image, 13192 samples per pixel of 16000
Octane: Refreshed image, 13324 samples per pixel of 16000
Octane: Refreshed image, 13455 samples per pixel of 16000
Octane: Refreshed image, 13587 samples per pixel of 16000
Octane: Refreshed image, 13720 samples per pixel of 16000
Octane: Refreshed image, 13850 samples per pixel of 16000
Octane: Refreshed image, 13985 samples per pixel of 16000
Octane: Refreshed image, 14115 samples per pixel of 16000
Octane: Refreshed image, 14247 samples per pixel of 16000
Octane: Refreshed image, 14381 samples per pixel of 16000
Octane: Refreshed image, 14513 samples per pixel of 16000
Octane: Refreshed image, 14644 samples per pixel of 16000
Octane: Refreshed image, 14777 samples per pixel of 16000
Octane: Refreshed image, 14910 samples per pixel of 16000
Octane: Refreshed image, 15039 samples per pixel of 16000
Octane: Refreshed image, 15172 samples per pixel of 16000
Octane: Refreshed image, 15305 samples per pixel of 16000
Octane: Refreshed image, 15435 samples per pixel of 16000
Octane: Refreshed image, 15568 samples per pixel of 16000
Octane: Refreshed image, 15701 samples per pixel of 16000
Octane: Refreshed image, 15832 samples per pixel of 16000
Octane: Refreshed image, 16000 samples per pixel of 16000
Octane: Frame 34 is rendered, saved "maya" image to "Z:/R2net/James_Allen/Render
ing/images/Ring_8825_Turn02b.034.iff"


Octane: starting animation of frame 35...
Octane: Scene scale factor: 1
0:00 Device "1: GeForce GTX 690 (compute model 3.0)" is DISABLED
0:01 Device "2: GeForce GTX 690 (compute model 3.0)" is DISABLED
0:01 Device "3: GeForce GTX 690 (compute model 3.0)" is DISABLED
0:01 Device "4: GeForce GTX 690 (compute model 3.0)" is DISABLED
0:01 Device "5: GeForce GTX 690 (compute model 3.0)" is DISABLED
0:01 Device "6: GeForce GTX 690 (compute model 3.0)" is DISABLED
0:01 Device "7: GeForce GTX 690 (compute model 3.0)" is DISABLED
0:01 Device "8: GeForce GTX 690 (compute model 3.0)" is DISABLED
Warning: Octane: No CUDA devices choosen to render on. Rendering on the second o
ne: GeForce GTX 690 (compute model 3.0)

[sampakinpa]

I tried to have another go rendering with one GPU and it froze again this time sooner on another sequence on frame 27.
I attached the verbose text file because I don't want to overcrowd this thread.

verbose06.04.13.txt.zip

(2.01 KiB) Downloaded 237 times

[JimStar]

It is definitely the problem outside the plugin - somewhere in the engine...
I will tell about it to the engine-team.

BTW, the only way I could enable one GPU to render, is by disabling all GPU's in the command line switches (-gpu1 0 -gpu2 0...) because those switches apparently support only 4 GPUs' and I have 8. so by disabling all 4 it actually disables all of the 8. then it select automatically one GPU as a default to render.

I will add the flags for more GPUs in the next version, thanks for reporting.

[sampakinpa]

Thanks JimStar for your help.
I finally succeed to batch render the sequences with only 2 GPU's (one GTX690) and it rendered all six files without any crash or freeze. what is going on? please tell me what do you think. is it a hardware or software problem?

[JimStar]

Thanks JimStar for your help.
I finally succeed to batch render the sequences with only 2 GPU's (one GTX690) and it rendered all six files without any crash or freeze. what is going on? please tell me what do you think. is it a hardware or software problem?

I think both...

[sampakinpa]

Now I tried to batch render one of the sequences (file size :27,733 KB, 160 frames, VRAM memory: 2GB) from inside maya using all 8 GPU's. maya exited the render on frame 38, no crash or freeze.
mayaRenderLog.txt indicates that 'Maya exited with status -1073740940'.

JimStar, I need help with that problem ASAP because we are in the process of changing our in-house Render pipeline from Maya-Mentalray to Maya-OctaneRender. our client is pressuring us to finish that already so this problem is a deal-breaker for us. I'll appreciate your help very much.
do you need more information from me like the scene files?

Sorry for my overtone.
Ethan.

render_info.zip

(22.01 KiB) Downloaded 235 times

[JimStar]

Now I tried to batch render one of the sequences (file size :27,733 KB, 160 frames, VRAM memory: 2GB) from inside maya using all 8 GPU's. maya exited the render on frame 38, no crash or freeze.
mayaRenderLog.txt indicates that 'Maya exited with status -1073740940'.

JimStar, I need help with that problem ASAP because we are in the process of changing our in-house Render pipeline from Maya-Mentalray to Maya-OctaneRender. our client is pressuring us to finish that already so this problem is a deal-breaker for us. I'll appreciate your help very much.
do you need more information from me like the scene files?

Sorry for my overtone.
Ethan.

The message "...done" at the end of log talks that the plugin has loaded all the scene to the engine, and just have started to wait for the image buffer... No problem occured in the plugin at all. And the message "Maya exited with status -1073740940" after it - is already not the plugin's message, it was printed just after the Octane engine module has crashed the plugin and the batch process itself...
So sorry, - so far I can't do here something (from the plugin's developer side) except telling about it to the engine team, and wait for the new SDK...
I think it is something happens at low-level during the scene-data reloading to the GPUs, as according to the logs - it always happens between the frames...

Thanks for your reports. We will try to catch it.
And yes, if it is possible - the scene may be helpful to reproduce this issue...

[JimStar]

sampakinpa
So, I've got some info from the engine-team...
What for version of drivers you have installed? There was the problem in previous versions of drivers with CUDA with more than 5 GPUs. So, if you have earlier drivers, try to install the last one and check if it will fix the problem...

[sampakinpa]

Yes I know about this problem, it was solved a month ago using the latest nvidia drivers installation.
Now we are using the latest nvidia drivers version 314.22 but it still crashes on batch render.

[sampakinpa]

Now Maya aplication crashed while I was on an IPR session.
here are the dump files:

WERD18A.tmp.zip

(4.88 MiB) Downloaded 238 times