Have been fighting for couple days now with different octane related pains. Optimizing the scene back and forth (to instances when got too little momery - that's what I thought then - and then from instances, back when generating instances broke sh*t on the other side). Now I'm in a place where I switched my 980Ti's to 1080Ti's for more VRAM and speed and all I get is Cuda error 700 all the time. On every device. Sometimes it's on the first frame I'm rendering, but most often on araound 30-50 frame which is just the worst. It appears it was the same error I got on 980Ti's that made me buy new cards :/
Have seen the problem on the forum before, but it was always a bit more specific (i.e. only one particular device failed) and there was never a good solution.
Tried OC'ing and default settings. Tried cooling it better with custom fan curves.
Right now, as I'm writing I'm trying out different octane versions, and maybe drivers - which version should I revert to best? EDIT: with 1080Ti I can't go back more than one drivers version back so the 373.06 mentioned in anohter post are a no go.
@Aoktar - I can send you the file prvately to snoop around - I'm pretty desperate here :/
All pc's have 3-4 of 1080Ti's, Win 7 and 32-64 GB RAM, latest test Octane and plugin.
Here's the example log:
Code: Select all
FRAME 2181 fps:30
MB:0/0 ST/MOV:0/1471 Nodes:3579 Tris:20.028m DispTris:22k Hairs:0 Meshes:1k
VRAM used/free/max:7.862Gb/2.68Gb/11Gb Out-of-core used:0Kb total used RAM:26.633Gb
MotBlurTM=0 sec. createTM=1.372 sec. matTM=0.126 sec. updateTM=24.108 sec.
Device:0 TotMem:11Gb rtData:508Mb film:54Mb geo:4.28Gb node:151Kb tex:3.032Gb unavailable:469Mb temperature:40
Device:1 TotMem:11Gb rtData:508Mb film:14Mb geo:4.28Gb node:151Kb tex:3.032Gb unavailable:315Mb temperature:45
Device:2 TotMem:11Gb rtData:508Mb film:14Mb geo:4.28Gb node:151Kb tex:3.032Gb unavailable:467Mb temperature:42
renderTM:15.575ms. totalTM:41.266ms.
FRAME 2182 fps:30
MB:0/0 ST/MOV:0/1471 Nodes:3579 Tris:20.028m DispTris:22k Hairs:0 Meshes:1k
VRAM used/free/max:7.862Gb/2.68Gb/11Gb Out-of-core used:0Kb total used RAM:26.622Gb
MotBlurTM=0 sec. createTM=1.372 sec. matTM=0.13 sec. updateTM=23.956 sec.
Device:0 TotMem:11Gb rtData:508Mb film:54Mb geo:4.28Gb node:151Kb tex:3.032Gb unavailable:469Mb temperature:41
Device:1 TotMem:11Gb rtData:508Mb film:14Mb geo:4.28Gb node:151Kb tex:3.032Gb unavailable:315Mb temperature:46
Device:2 TotMem:11Gb rtData:508Mb film:14Mb geo:4.28Gb node:151Kb tex:3.032Gb unavailable:467Mb temperature:43
CUDA error 700 on device 0: an illegal memory access was encountered
CUDA error 700 on device 1: an illegal memory access was encountered
CUDA error 700 on device 2: an illegal memory access was encountered
-> failed to launch kernel(ptBrdf2)
-> kernel execution failed(report)
-> failed to launch kernel(ptTrace1m)
CUDA error 700 on device 2: an illegal memory access was encountered
-> failed to launch kernel(ptBrdf2)
device 0: path tracing kernel failed
device 0: detected an error on render device! trying to recover...
device 1: path tracing kernel failed
CUDA error 700 on device 0: an illegal memory access was encountered
device 1: detected an error on render device! trying to recover...
-> failed to deallocate device memory
CUDA error 700 on device 0: an illegal memory access was encountered
-> could not get memory info
device 2: path tracing kernel failed
CUDA error 700 on device 0: an illegal memory access was encountered
-> failed to deallocate device memory
CUDA error 700 on device 0: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 1: an illegal memory access was encountered
CUDA error 700 on device 0: an illegal memory access was encountered
-> failed to deallocate device memory
device 2: detected an error on render device! trying to recover...
CUDA error 700 on device 2: an illegal memory access was encountered
-> failed to deallocate device memory
CUDA error 700 on device 2: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 2: an illegal memory access was encountered
-> failed to deallocate device memory
CUDA error 700 on device 2: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 2: an illegal memory access was encountered
-> failed to deallocate device memory
CUDA error 700 on device 2: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 2: an illegal memory access was encountered
-> failed to deallocate device memory
CUDA error 700 on device 2: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 2: an illegal memory access was encountered
-> failed to deallocate device memory
CUDA error 700 on device 2: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 2: an illegal memory access was encountered
-> failed to deallocate device memory
CUDA error 700 on device 0: an illegal memory access was encountered
CUDA error 700 on device 2: an illegal memory access was encountered
-> could not get memory info
-> could not get memory info
CUDA error 700 on device 2: an illegal memory access was encountered
-> failed to deallocate device memory
CUDA error 700 on device 0: an illegal memory access was encountered
-> failed to deallocate device memory
CUDA error 700 on device 0: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 2: an illegal memory access was encountered
-> failed to deallocate device memory
CUDA error 700 on device 0: an illegal memory access was encountered
-> failed to deallocate device memory
CUDA error 700 on device 0: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 0: an illegal memory access was encountered
-> failed to deallocate device memory
-> could not get memory info
CUDA error 700 on device 0: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 2: an illegal memory access was encountered
CUDA error 700 on device 1: an illegal memory access was encountered
-> failed to deallocate device memory
-> could not get memory info
CUDA error 700 on device 0: an illegal memory access was encountered
-> failed to deallocate device memory
CUDA error 700 on device 1: an illegal memory access was encountered
-> failed to deallocate device memory
CUDA error 700 on device 2: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 2: an illegal memory access was encountered
-> could not get memory info
-> could not get memory info
CUDA error 700 on device 2: an illegal memory access was encountered
-> failed to deallocate device array
CUDA error 700 on device 0: an illegal memory access was encountered
CUDA error 700 on device 2: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 2: an illegal memory access was encountered
-> failed to deallocate device array
-> failed to deallocate device array
CUDA error 700 on device 2: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 0: an illegal memory access was encountered
CUDA error 700 on device 2: an illegal memory access was encountered
-> failed to deallocate device array
-> failed to deallocate device array
-> could not get memory info
CUDA error 700 on device 1: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 0: an illegal memory access was encountered
-> failed to deallocate device array
CUDA error 700 on device 1: an illegal memory access was encountered
-> failed to deallocate device array
CUDA error 700 on device 0: an illegal memory access was encountered
CUDA error 700 on device 2: an illegal memory access was encountered
CUDA error 700 on device 1: an illegal memory access was encountered
-> could not get memory info
-> could not get memory info
CUDA error 700 on device 1: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 0: an illegal memory access was encountered
-> failed to deallocate device array
-> failed to deallocate device array
CUDA error 700 on device 2: an illegal memory access was encountered
-> failed to deallocate device array
CUDA error 700 on device 0: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 1: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 2: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 1: an illegal memory access was encountered
-> failed to deallocate device array
CUDA error 700 on device 0: an illegal memory access was encountered
-> failed to deallocate device array
CUDA error 700 on device 1: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 2: an illegal memory access was encountered
-> failed to deallocate device array
CUDA error 700 on device 2: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 0: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 2: an illegal memory access was encountered
-> failed to deallocate device array
CUDA error 700 on device 0: an illegal memory access was encountered
-> failed to deallocate device array
CUDA error 700 on device 2: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 0: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 0: an illegal memory access was encountered
-> failed to deallocate device array
CUDA error 700 on device 0: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 1: an illegal memory access was encountered
-> failed to deallocate device array
CUDA error 700 on device 0: an illegal memory access was encountered
-> failed to deallocate device array
CUDA error 700 on device 2: an illegal memory access was encountered
-> failed to deallocate device array
CUDA error 700 on device 2: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 1: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 2: an illegal memory access was encountered
-> failed to deallocate device array
CUDA error 700 on device 0: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 1: an illegal memory access was encountered
-> failed to deallocate device array
CUDA error 700 on device 0: an illegal memory access was encountered
CUDA error 700 on device 2: an illegal memory access was encountered
CUDA error 700 on device 1: an illegal memory access was encountered
-> could not get memory info
-> failed to deallocate device array
CUDA error 700 on device 1: an illegal memory access was encountered
-> failed to deallocate device array
CUDA error 700 on device 0: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 1: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 0: an illegal memory access was encountered
-> could not get memory info
-> failed to deallocate device array
CUDA error 700 on device 2: an illegal memory access was encountered
-> failed to deallocate device array
CUDA error 700 on device 0: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 2: an illegal memory access was encountered
CUDA error 700 on device 0: an illegal memory access was encountered
CUDA error 700 on device 1: an illegal memory access was encountered
-> failed to deallocate device array
-> could not get memory info
-> failed to deallocate device array
CUDA error 700 on device 1: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 2: an illegal memory access was encountered
-> failed to deallocate device array
CUDA error 700 on device 1: an illegal memory access was encountered
CUDA error 700 on device 0: an illegal memory access was encountered
-> failed to deallocate device array
CUDA error 700 on device 2: an illegal memory access was encountered
-> could not get memory info
-> could not get memory info
CUDA error 700 on device 1: an illegal memory access was encountered
-> failed to bind device to current thread
device 1: failed to initialize resources
device 1: failed to recover - giving up and switching device to failed
----------------------------------------------------------------------------------------
<<< Render failure detected!!! >>>
Please check render statistics to solve the problem.
Device:0 TotMem:11Gb rtData:0Kb film:0Kb geo:0Kb node:0Kb tex:0Kb unavailable:8.32Gb temperature:41
Device:1 TotMem:11Gb rtData:0Kb film:0Kb geo:0Kb node:0Kb tex:0Kb unavailable:8.13Gb temperature:46
Device:2 TotMem:11Gb rtData:0Kb film:0Kb geo:0Kb node:0Kb tex:0Kb unavailable:8.279Gb temperature:43
Check geometry subdivisons.
Picture viewer uses 'Render' subdivisions. But Live Viewer uses 'Editor' subdivision values.
Try to decrease 'render/editor' subdivison values.
CUDA error 700 on device 0: an illegal memory access was encountered
-> failed to bind device to current thread
device 0: failed to initialize resources
device 0: failed to recover - giving up and switching device to failed
CUDA error 700 on device 2: an illegal memory access was encountered
-> failed to bind device to current thread
device 2: failed to initialize resources
device 2: failed to recover - giving up and switching device to failed
renderTM:3.306ms. totalTM:28.856ms.
Break rendering...
Tried to access null node
Tried to access pin via an invalid ID P_MESH (111)
Tried to access null node pin
Tried to access null node
Tried to access pin via an invalid ID P_MESH (111)
Tried to access null node pin
Could not load 'E:\Google Drive\PROJEKTY\2017.03_SAMSUNG_VML_Missed_Spaceflight\c4d\animatic\tex\xxxx_animacja_lewy_panel_maska_druga\xxxx_animacja_lewy_panel_maska_druga2000.png:rgba'.
Could not load 'E:\Google Drive\PROJEKTY\2017.03_SAMSUNG_VML_Missed_Spaceflight\c4d\animatic\tex\zamarzanie_mini\ice_mini_bw_6_2000.jpg:rgba'.
Could not load 'E:\Google Drive\PROJEKTY\2017.03_SAMSUNG_VML_Missed_Spaceflight\c4d\animatic\tex\xxxx_animacja_lewy_panel_maska_druga\xxxx_animacja_lewy_panel_maska_druga2000.png:rgba'.
Could not load 'E:\Google Drive\PROJEKTY\2017.03_SAMSUNG_VML_Missed_Spaceflight\c4d\animatic\tex\xxxx_animacja_lewy_panel_maska_druga\xxxx_animacja_lewy_panel_maska_druga2000.png:rgba'.
Could not load 'E:\Google Drive\PROJEKTY\2017.03_SAMSUNG_VML_Missed_Spaceflight\c4d\animatic\tex\xxxx_animacja_lewy_panel_maska\xxxx_animacja_lewy_panel_maska2000.png:rgba'.
Could not load 'E:\Google Drive\PROJEKTY\2017.03_SAMSUNG_VML_Missed_Spaceflight\c4d\animatic\tex\zamarzanie_duze\ice_big_bw_5m_2000.png:rgba'.
Could not load 'E:\Google Drive\PROJEKTY\2017.03_SAMSUNG_VML_Missed_Spaceflight\c4d\animatic\tex\zamarzanie_mini\ice_mini_bw_6_2000.jpg:rgba'.
--------------------------- EXPORT LOG ---------------------------
CUDA error 700 on device 0: an illegal memory access was encountered
-> failed to bind device to current thread
device 0: failed to initialize resources
device 0: detected an error on render device! trying to recover...
CUDA error 700 on device 0: an illegal memory access was encountered
-> failed to bind device to current thread
device 0: failed to initialize resources
device 0: failed to recover - giving up and switching device to failed
CUDA error 700 on device 1: an illegal memory access was encountered
-> failed to bind device to current thread
device 1: failed to initialize resources
device 1: detected an error on render device! trying to recover...
CUDA error 700 on device 1: an illegal memory access was encountered
-> failed to bind device to current thread
device 1: failed to initialize resources
device 1: failed to recover - giving up and switching device to failed
CUDA error 700 on device 2: an illegal memory access was encountered
-> failed to bind device to current thread
device 2: failed to initialize resources
device 2: detected an error on render device! trying to recover...
----------------------------------------------------------------------------------------
<<< Render failure detected!!! >>>
Please check render statistics to solve the problem.
CUDA error 700 on device 2: an illegal memory access was encountered
-> failed to bind device to current thread
device 2: failed to initialize resources
device 2: failed to recover - giving up and switching device to failed
MB:0/0 ST/MOV:0/1471 Nodes:324 Tris:0 DispTris:0 Hairs:0 Meshes:0
Device:0 TotMem:11Gb rtData:0Kb film:0Kb geo:0Kb node:0Kb tex:0Kb unavailable:8.32Gb temperature:27
Device:1 TotMem:11Gb rtData:0Kb film:0Kb geo:0Kb node:0Kb tex:0Kb unavailable:8.13Gb temperature:27
Device:2 TotMem:11Gb rtData:0Kb film:0Kb geo:0Kb node:0Kb tex:0Kb unavailable:8.279Gb temperature:28
VRAM used/free/max:0Kb/0Kb/0Kb
Free VRAM is too low!
Try to decrease polygon counts and use out-of-core for textures.
Use 'render instances' when it's possible.'
Export materials time= 4404.775 ms
Collect objects time= 36.468 ms
Mesh creation time = 10741.391 ms.
MB:0/0 ST/MOV:0/1471 Nodes:3515 Tris:0 DispTris:0 Hairs:0 Meshes:0
Total export Time = 15206.179 ms
