How can I avoid a CUDA error 700?

[helloimneil]

I'm working on an animation project (for my own pleasure) and this is the second time I've encountered this CUDA error 700 nonsense. The first time the render stopped midway through and per the Octane log's recommendation I tried to decrease the number of subdivisions in my scene. After I deleted the subdivision on an object I tried re-rendering and the render completed all of the frames.

I rendered another small clip yesterday and no problems at all.

But then... tonight, I began rendering a third clip and encountered a render failure which caused it to stop way, way short of where it needed to finish at.
The below is from the log Octane made (I have the entirety of the log if anyone would like to see that).

What am I doing wrong with Octane? I only have one object with a cloth subdivision on it to add a little thickness to the visible sides.

This is the 3rd time I've encountered this error and, to be honest, makes me really nervous about accepting any large scale projects since the reliability of my renders is currently at like... 50%.

Seriously, any help from you guys would be super appreciated because as of now I'm at a total loss as to A) what this error means, and B) how to prevent this from happening.

<<< Render failure detected!!! >>>
Please check render statistics to solve the problem.
Device:0 TotMem:8Gb rtData:0Kb film:0Kb geo:0Kb node:0Kb tex:0Kb unavailable:1.812Gb temperature:57
Device:1 TotMem:8Gb rtData:0Kb film:0Kb geo:0Kb node:0Kb tex:0Kb unavailable:1.812Gb temperature:68
Device:2 TotMem:8Gb rtData:0Kb film:0Kb geo:0Kb node:0Kb tex:0Kb unavailable:2.014Gb temperature:64
Check geometry subdivisons.
Picture viewer uses 'Render' subdivisions. But Live Viewer uses 'Editor' subdivision values.
Try to decrease 'render/editor' subdivison values.
CUDA error 700 on device 0: an illegal memory access was encountered
-> failed to bind device to current thread
device 0: failed to initialize resources
device 0: failed to recover - giving up and switching device to failed
CUDA error 700 on device 1: an illegal memory access was encountered
-> failed to bind device to current thread
device 1: failed to initialize resources
device 1: failed to recover - giving up and switching device to failed
Break rendering...

[abstrax]

Which version are you using?

[helloimneil]

Of Octane or of Cinema 4D?

I'm using the most recent version of Octane, and as for C4D I'm using R17.

[aoktar]

Of Octane or of Cinema 4D?

I'm using the most recent version of Octane, and as for C4D I'm using R17.

What's numbers? 3.06 stable or test versions? Windows/OSX? GPU information? Drivers?

[helloimneil]

I'm using Octane (C4D plugin) version 3.05.3, I did not realize there was a 3.06 build.
I'm using the stable version.

My OS is Windows 10.

I have 3x GTX 1070 FEs, my drivers are up to date: version 3.5.0.70

[aoktar]

I'm using Octane (C4D plugin) version 3.05.3, I did not realize there was a 3.06 build.
I'm using the stable version.

My OS is Windows 10.

I have 3x GTX 1070 FEs, my drivers are up to date: version 3.5.0.70

Can you please check it with 3.06?
viewtopic.php?f=85&t=60816

[helloimneil]

aoktar! I think my issues have been solved.

I think the problem was a combination of not having the latest version of R17-- I had 0.48, the newest version of R17 is 0.55, could've sworn I had the most recent version, and also not having the current version of Octane.

As of now both softwares are up to date. The scene has been re-rendered with no CUDA error! Hopefully this trend continues!

aoktar, again, thank you for the help, I sincerely appreciate it. Should we ever meet in real life I'll gladly buy you a drink or some lunch/dinner. Cheers!

[aoktar]

Glad to hear that! I'll not forget this offer

[helloimneil]

aoktar, I may have you on the hook for a potential second drink/lunch...

I began rendering a new clip from the same animation project only to encounter the dreaded (and confounding) CUDA error 700.
So, I tried rendering the scene from where it stopped short and, yet again, I encountered a render failure

Here is an excerpt from the log:

(Frame 226 and 227 rendered as expected, but then, at frame 228, the error is encountered)

FRAME 226 fps:30
MB:0/0 ST/MOV:0/14 Nodes:52 Tris:114k DispTris:0 Hairs:0 Meshes:15
VRAM used/free/max:362Mb/6.076Gb/8Gb Out-of-core used:0Kb total used RAM:7.375Gb
MotBlurTM=0 sec. createTM=0.036 sec. matTM=0 sec. updateTM=0.564 sec.
Device:0 TotMem:8Gb rtData:284Mb film:10Mb geo:24Mb node:9Kb tex:42Mb unavailable:1.357Gb temperature:54
Device:1 TotMem:8Gb rtData:284Mb film:10Mb geo:24Mb node:9Kb tex:42Mb unavailable:1.357Gb temperature:66
Device:2 TotMem:8Gb rtData:284Mb film:10Mb geo:24Mb node:9Kb tex:42Mb unavailable:1.57Gb temperature:60
renderTM:115.805ms. totalTM:116.405ms.


FRAME 227 fps:30
MB:0/0 ST/MOV:0/14 Nodes:52 Tris:115k DispTris:0 Hairs:0 Meshes:15
VRAM used/free/max:362Mb/6.076Gb/8Gb Out-of-core used:0Kb total used RAM:7.4Gb
MotBlurTM=0 sec. createTM=0.04 sec. matTM=0 sec. updateTM=0.528 sec.
Device:0 TotMem:8Gb rtData:284Mb film:10Mb geo:25Mb node:9Kb tex:42Mb unavailable:1.357Gb temperature:52
Device:1 TotMem:8Gb rtData:284Mb film:10Mb geo:25Mb node:9Kb tex:42Mb unavailable:1.357Gb temperature:65
Device:2 TotMem:8Gb rtData:284Mb film:10Mb geo:25Mb node:9Kb tex:42Mb unavailable:1.57Gb temperature:60
renderTM:116.052ms. totalTM:116.62ms.


FRAME 228 fps:30
MB:0/0 ST/MOV:0/14 Nodes:52 Tris:114k DispTris:0 Hairs:0 Meshes:15
VRAM used/free/max:362Mb/6.076Gb/8Gb Out-of-core used:0Kb total used RAM:7.397Gb
MotBlurTM=0 sec. createTM=0.036 sec. matTM=0 sec. updateTM=0.539 sec.
Device:0 TotMem:8Gb rtData:284Mb film:10Mb geo:24Mb node:9Kb tex:42Mb unavailable:1.357Gb temperature:52
Device:1 TotMem:8Gb rtData:284Mb film:10Mb geo:24Mb node:9Kb tex:42Mb unavailable:1.357Gb temperature:65
Device:2 TotMem:8Gb rtData:284Mb film:10Mb geo:24Mb node:9Kb tex:42Mb unavailable:1.57Gb temperature:60
CUDA error 700 on device 1: an illegal memory access was encountered
-> kernel execution failed(report)
CUDA error 700 on device 1: an illegal memory access was encountered
-> failed to copy memory from device.
device 1: path tracing kernel failed
device 1: detected an error on render device! trying to recover...
CUDA error 700 on device 1: an illegal memory access was encountered
-> failed to deallocate device memory
CUDA error 700 on device 1: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 1: an illegal memory access was encountered
-> failed to deallocate device memory
CUDA error 700 on device 1: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 1: an illegal memory access was encountered
-> failed to deallocate device memory
CUDA error 700 on device 1: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 1: an illegal memory access was encountered
-> failed to deallocate device memory
CUDA error 700 on device 1: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 1: an illegal memory access was encountered
-> failed to deallocate device memory
CUDA error 700 on device 1: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 1: an illegal memory access was encountered
-> failed to deallocate device memory
CUDA error 700 on device 1: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 1: an illegal memory access was encountered
-> failed to deallocate device memory
CUDA error 700 on device 1: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 1: an illegal memory access was encountered
-> failed to deallocate device memory
CUDA error 700 on device 1: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 1: an illegal memory access was encountered
-> failed to deallocate device memory
CUDA error 700 on device 1: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 1: an illegal memory access was encountered
-> failed to deallocate device memory
CUDA error 700 on device 1: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 1: an illegal memory access was encountered
-> failed to deallocate device memory
CUDA error 700 on device 1: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 1: an illegal memory access was encountered
-> failed to deallocate device memory
CUDA error 700 on device 1: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 1: an illegal memory access was encountered
-> failed to deallocate device memory
CUDA error 700 on device 1: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 1: an illegal memory access was encountered
-> failed to deallocate device memory
CUDA error 700 on device 1: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 1: an illegal memory access was encountered
-> failed to deallocate device memory
CUDA error 700 on device 1: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 1: an illegal memory access was encountered
-> failed to deallocate device memory
CUDA error 700 on device 1: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 1: an illegal memory access was encountered
-> failed to deallocate device memory
CUDA error 700 on device 1: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 1: an illegal memory access was encountered
-> failed to deallocate device memory
CUDA error 700 on device 1: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 1: an illegal memory access was encountered
-> failed to deallocate device memory
CUDA error 700 on device 1: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 1: an illegal memory access was encountered
-> failed to deallocate device memory
CUDA error 700 on device 1: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 1: an illegal memory access was encountered
-> failed to deallocate device array
CUDA error 700 on device 1: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 1: an illegal memory access was encountered
-> failed to deallocate device array
CUDA error 700 on device 1: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 1: an illegal memory access was encountered
-> failed to deallocate device array
CUDA error 700 on device 1: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 1: an illegal memory access was encountered
-> failed to deallocate device array
CUDA error 700 on device 1: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 1: an illegal memory access was encountered
-> failed to deallocate device array
CUDA error 700 on device 1: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 1: an illegal memory access was encountered
-> failed to deallocate device array
CUDA error 700 on device 1: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 1: an illegal memory access was encountered
-> failed to deallocate device array
CUDA error 700 on device 1: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 1: an illegal memory access was encountered
-> failed to deallocate device array
CUDA error 700 on device 1: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 1: an illegal memory access was encountered
-> failed to deallocate device array
CUDA error 700 on device 1: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 1: an illegal memory access was encountered
-> failed to deallocate device array
CUDA error 700 on device 1: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 1: an illegal memory access was encountered
-> failed to deallocate device array
CUDA error 700 on device 1: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 1: an illegal memory access was encountered
-> failed to deallocate device array
CUDA error 700 on device 1: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 1: an illegal memory access was encountered
-> failed to deallocate device array
CUDA error 700 on device 1: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 1: an illegal memory access was encountered
-> failed to deallocate device memory
CUDA error 700 on device 1: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 1: an illegal memory access was encountered
-> failed to deallocate pinned memory
CUDA error 700 on device 1: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 1: an illegal memory access was encountered
-> failed to deallocate pinned memory
CUDA error 700 on device 1: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 1: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 1: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 1: an illegal memory access was encountered
-> failed to deallocate device memory
CUDA error 700 on device 1: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 1: an illegal memory access was encountered
-> failed to deallocate pinned memory
CUDA error 700 on device 1: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 1: an illegal memory access was encountered
-> failed to deallocate pinned memory
CUDA error 700 on device 1: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 1: an illegal memory access was encountered
-> failed to deallocate device memory
CUDA error 700 on device 1: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 1: an illegal memory access was encountered
-> failed to deallocate device memory
CUDA error 700 on device 1: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 1: an illegal memory access was encountered
-> failed to deallocate device memory
CUDA error 700 on device 1: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 1: an illegal memory access was encountered
-> failed to deallocate device memory
CUDA error 700 on device 1: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 1: an illegal memory access was encountered
-> failed to deallocate device memory
CUDA error 700 on device 1: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 1: an illegal memory access was encountered
-> failed to deallocate device memory
CUDA error 700 on device 1: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 1: an illegal memory access was encountered
-> failed to deallocate device memory
CUDA error 700 on device 1: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 1: an illegal memory access was encountered
-> failed to deallocate device memory
CUDA error 700 on device 1: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 1: an illegal memory access was encountered
-> failed to deallocate device memory
CUDA error 700 on device 1: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 1: an illegal memory access was encountered
-> failed to deallocate device memory
CUDA error 700 on device 1: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 1: an illegal memory access was encountered
-> failed to deallocate device memory
CUDA error 700 on device 1: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 1: an illegal memory access was encountered
-> failed to deallocate device memory
CUDA error 700 on device 1: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 1: an illegal memory access was encountered
-> failed to deallocate device memory
CUDA error 700 on device 1: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 1: an illegal memory access was encountered
-> failed to deallocate device memory
CUDA error 700 on device 1: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 1: an illegal memory access was encountered
-> failed to unload module
CUDA error 700 on device 2: an illegal memory access was encountered
-> kernel execution failed(report)
CUDA error 700 on device 2: an illegal memory access was encountered
-> failed to launch kernel(ptBrdf2)
device 2: path tracing kernel failed
device 2: detected an error on render device! trying to recover...
CUDA error 700 on device 2: an illegal memory access was encountered
-> failed to deallocate device memory
CUDA error 700 on device 2: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 2: an illegal memory access was encountered
-> failed to deallocate device memory
CUDA error 700 on device 2: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 2: an illegal memory access was encountered
-> failed to deallocate device memory
CUDA error 700 on device 2: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 2: an illegal memory access was encountered
-> failed to deallocate device memory
CUDA error 700 on device 2: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 2: an illegal memory access was encountered
-> failed to deallocate device memory
CUDA error 700 on device 2: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 2: an illegal memory access was encountered
-> failed to deallocate device memory
CUDA error 700 on device 2: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 2: an illegal memory access was encountered
-> failed to deallocate device memory
CUDA error 700 on device 2: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 2: an illegal memory access was encountered
-> failed to deallocate device memory
CUDA error 700 on device 2: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 2: an illegal memory access was encountered
-> failed to deallocate device memory
CUDA error 700 on device 2: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 2: an illegal memory access was encountered
-> failed to deallocate device memory
CUDA error 700 on device 2: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 2: an illegal memory access was encountered
-> failed to deallocate device memory
CUDA error 700 on device 2: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 2: an illegal memory access was encountered
-> failed to deallocate device memory
CUDA error 700 on device 2: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 2: an illegal memory access was encountered
-> failed to deallocate device memory
CUDA error 700 on device 2: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 2: an illegal memory access was encountered
-> failed to deallocate device memory
CUDA error 700 on device 2: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 2: an illegal memory access was encountered
-> failed to deallocate device memory
CUDA error 700 on device 2: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 2: an illegal memory access was encountered
-> failed to deallocate device memory
CUDA error 700 on device 2: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 2: an illegal memory access was encountered
-> failed to deallocate device memory
CUDA error 700 on device 2: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 2: an illegal memory access was encountered
-> failed to deallocate device memory
CUDA error 700 on device 2: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 2: an illegal memory access was encountered
-> failed to deallocate device memory
CUDA error 700 on device 2: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 2: an illegal memory access was encountered
-> failed to deallocate device memory
CUDA error 700 on device 2: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 2: an illegal memory access was encountered
-> failed to deallocate device array
CUDA error 700 on device 2: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 2: an illegal memory access was encountered
-> failed to deallocate device array
CUDA error 700 on device 2: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 2: an illegal memory access was encountered
-> failed to deallocate device array
CUDA error 700 on device 2: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 2: an illegal memory access was encountered
-> failed to deallocate device array
CUDA error 700 on device 2: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 2: an illegal memory access was encountered
-> failed to deallocate device array
CUDA error 700 on device 2: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 2: an illegal memory access was encountered
-> failed to deallocate device array
CUDA error 700 on device 2: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 2: an illegal memory access was encountered
-> failed to deallocate device array
CUDA error 700 on device 2: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 2: an illegal memory access was encountered
-> failed to deallocate device array
CUDA error 700 on device 2: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 2: an illegal memory access was encountered
-> failed to deallocate device array
CUDA error 700 on device 2: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 2: an illegal memory access was encountered
-> failed to deallocate device array
CUDA error 700 on device 2: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 2: an illegal memory access was encountered
-> failed to deallocate device array
CUDA error 700 on device 2: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 2: an illegal memory access was encountered
-> failed to deallocate device array
CUDA error 700 on device 2: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 2: an illegal memory access was encountered
-> failed to deallocate device array
CUDA error 700 on device 2: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 2: an illegal memory access was encountered
-> failed to deallocate device memory
CUDA error 700 on device 2: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 2: an illegal memory access was encountered
-> failed to deallocate pinned memory
CUDA error 700 on device 2: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 2: an illegal memory access was encountered
-> failed to deallocate pinned memory
CUDA error 700 on device 2: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 2: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 2: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 2: an illegal memory access was encountered
-> failed to deallocate device memory
CUDA error 700 on device 2: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 2: an illegal memory access was encountered
-> failed to deallocate pinned memory
CUDA error 700 on device 2: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 2: an illegal memory access was encountered
-> failed to deallocate pinned memory
CUDA error 700 on device 2: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 2: an illegal memory access was encountered
-> failed to deallocate device memory
CUDA error 700 on device 2: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 2: an illegal memory access was encountered
-> failed to deallocate device memory
CUDA error 700 on device 2: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 2: an illegal memory access was encountered
-> failed to deallocate device memory
CUDA error 700 on device 2: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 2: an illegal memory access was encountered
-> failed to deallocate device memory
CUDA error 700 on device 2: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 2: an illegal memory access was encountered
-> failed to deallocate device memory
CUDA error 700 on device 2: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 2: an illegal memory access was encountered
-> failed to deallocate device memory
CUDA error 700 on device 2: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 2: an illegal memory access was encountered
-> failed to deallocate device memory
CUDA error 700 on device 2: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 2: an illegal memory access was encountered
-> failed to deallocate device memory
CUDA error 700 on device 2: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 2: an illegal memory access was encountered
-> failed to deallocate device memory
CUDA error 700 on device 2: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 2: an illegal memory access was encountered
-> failed to deallocate device memory
CUDA error 700 on device 2: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 2: an illegal memory access was encountered
-> failed to deallocate device memory
CUDA error 700 on device 2: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 2: an illegal memory access was encountered
-> failed to deallocate device memory
CUDA error 700 on device 2: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 2: an illegal memory access was encountered
-> failed to deallocate device memory
CUDA error 700 on device 2: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 2: an illegal memory access was encountered
-> failed to deallocate device memory
CUDA error 700 on device 2: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 2: an illegal memory access was encountered
-> failed to unload module
CUDA error 700 on device 0: an illegal memory access was encountered
-> kernel execution failed(report)
CUDA error 700 on device 0: an illegal memory access was encountered
-> failed to launch kernel(ptBrdf2)
device 0: path tracing kernel failed
device 0: detected an error on render device! trying to recover...
CUDA error 700 on device 0: an illegal memory access was encountered
-> failed to deallocate device memory
CUDA error 700 on device 0: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 0: an illegal memory access was encountered
-> failed to deallocate device memory
CUDA error 700 on device 0: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 0: an illegal memory access was encountered
-> failed to deallocate device memory
CUDA error 700 on device 0: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 0: an illegal memory access was encountered
-> failed to deallocate device memory
CUDA error 700 on device 0: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 0: an illegal memory access was encountered
-> failed to deallocate device memory
CUDA error 700 on device 0: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 0: an illegal memory access was encountered
-> failed to deallocate device memory
CUDA error 700 on device 0: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 0: an illegal memory access was encountered
-> failed to deallocate device memory
CUDA error 700 on device 0: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 0: an illegal memory access was encountered
-> failed to deallocate device memory
CUDA error 700 on device 0: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 0: an illegal memory access was encountered
-> failed to deallocate device memory
CUDA error 700 on device 0: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 0: an illegal memory access was encountered
-> failed to deallocate device memory
CUDA error 700 on device 0: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 0: an illegal memory access was encountered
-> failed to deallocate device memory
CUDA error 700 on device 0: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 0: an illegal memory access was encountered
-> failed to deallocate device memory
CUDA error 700 on device 0: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 0: an illegal memory access was encountered
-> failed to deallocate device memory
CUDA error 700 on device 0: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 0: an illegal memory access was encountered
-> failed to deallocate device memory
CUDA error 700 on device 0: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 0: an illegal memory access was encountered
-> failed to deallocate device memory
CUDA error 700 on device 0: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 0: an illegal memory access was encountered
-> failed to deallocate device memory
CUDA error 700 on device 0: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 0: an illegal memory access was encountered
-> failed to deallocate device memory
CUDA error 700 on device 0: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 0: an illegal memory access was encountered
-> failed to deallocate device memory
CUDA error 700 on device 0: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 0: an illegal memory access was encountered
-> failed to deallocate device memory
CUDA error 700 on device 0: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 0: an illegal memory access was encountered
-> failed to deallocate device memory
CUDA error 700 on device 0: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 0: an illegal memory access was encountered
-> failed to deallocate device array
CUDA error 700 on device 0: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 0: an illegal memory access was encountered
-> failed to deallocate device array
CUDA error 700 on device 0: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 0: an illegal memory access was encountered
-> failed to deallocate device array
CUDA error 700 on device 0: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 0: an illegal memory access was encountered
-> failed to deallocate device array
CUDA error 700 on device 0: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 0: an illegal memory access was encountered
-> failed to deallocate device array
CUDA error 700 on device 0: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 0: an illegal memory access was encountered
-> failed to deallocate device array
CUDA error 700 on device 0: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 0: an illegal memory access was encountered
-> failed to deallocate device array
CUDA error 700 on device 0: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 0: an illegal memory access was encountered
-> failed to deallocate device array
CUDA error 700 on device 0: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 0: an illegal memory access was encountered
-> failed to deallocate device array
CUDA error 700 on device 0: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 0: an illegal memory access was encountered
-> failed to deallocate device array
CUDA error 700 on device 0: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 0: an illegal memory access was encountered
-> failed to deallocate device array
CUDA error 700 on device 0: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 0: an illegal memory access was encountered
-> failed to deallocate device array
CUDA error 700 on device 0: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 0: an illegal memory access was encountered
-> failed to deallocate device array
CUDA error 700 on device 0: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 0: an illegal memory access was encountered
-> failed to deallocate device memory
CUDA error 700 on device 0: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 0: an illegal memory access was encountered
-> failed to deallocate pinned memory
CUDA error 700 on device 0: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 0: an illegal memory access was encountered
-> failed to deallocate pinned memory
CUDA error 700 on device 0: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 0: an illegal memory access was encountered
-> failed to unregister pinned memory
CUDA error 700 on device 0: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 0: an illegal memory access was encountered
-> failed to unregister pinned memory
CUDA error 700 on device 0: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 0: an illegal memory access was encountered
-> failed to deallocate device memory
CUDA error 700 on device 0: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 0: an illegal memory access was encountered
-> failed to deallocate pinned memory
CUDA error 700 on device 0: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 0: an illegal memory access was encountered
-> failed to deallocate pinned memory
CUDA error 700 on device 0: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 0: an illegal memory access was encountered
-> failed to deallocate device memory
CUDA error 700 on device 0: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 0: an illegal memory access was encountered
-> failed to deallocate device memory
CUDA error 700 on device 0: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 0: an illegal memory access was encountered
-> failed to deallocate device memory
CUDA error 700 on device 0: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 0: an illegal memory access was encountered
-> failed to deallocate device memory
CUDA error 700 on device 0: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 0: an illegal memory access was encountered
-> failed to deallocate device memory
CUDA error 700 on device 0: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 0: an illegal memory access was encountered
-> failed to deallocate device memory
CUDA error 700 on device 0: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 0: an illegal memory access was encountered
-> failed to deallocate device memory
CUDA error 700 on device 0: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 0: an illegal memory access was encountered
-> failed to deallocate device memory
CUDA error 700 on device 0: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 0: an illegal memory access was encountered
-> failed to deallocate device memory
CUDA error 700 on device 0: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 0: an illegal memory access was encountered
-> failed to deallocate device memory
CUDA error 700 on device 0: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 0: an illegal memory access was encountered
-> failed to deallocate device memory
CUDA error 700 on device 0: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 0: an illegal memory access was encountered
-> failed to deallocate device memory
CUDA error 700 on device 0: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 0: an illegal memory access was encountered
-> failed to deallocate device memory
CUDA error 700 on device 0: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 0: an illegal memory access was encountered
-> failed to deallocate device memory
CUDA error 700 on device 0: an illegal memory access was encountered
-> could not get memory info
CUDA error 700 on device 0: an illegal memory access was encountered
-> failed to unload module
CUDA error 700 on device 2: an illegal memory access was encountered
-> failed to bind device to current thread
device 2: failed to initialize resources
device 2: failed to recover - giving up and switching device to failed
----------------------------------------------------------------------------------------
<<< Render failure detected!!! >>>
Please check render statistics to solve the problem.
Device:0 TotMem:8Gb rtData:0Kb film:0Kb geo:0Kb node:0Kb tex:0Kb unavailable:1.71Gb temperature:56
Device:1 TotMem:8Gb rtData:0Kb film:0Kb geo:0Kb node:0Kb tex:0Kb unavailable:1.71Gb temperature:67
Device:2 TotMem:8Gb rtData:0Kb film:0Kb geo:0Kb node:0Kb tex:0Kb unavailable:1.924Gb temperature:63
Check geometry subdivisons.
Picture viewer uses 'Render' subdivisions. But Live Viewer uses 'Editor' subdivision values.
Try to decrease 'render/editor' subdivison values.
CUDA error 700 on device 0: an illegal memory access was encountered
-> failed to bind device to current thread
device 0: failed to initialize resources
device 0: failed to recover - giving up and switching device to failed
CUDA error 700 on device 1: an illegal memory access was encountered
-> failed to bind device to current thread
device 1: failed to initialize resources
device 1: failed to recover - giving up and switching device to failed
renderTM:106.851ms. totalTM:107.426ms.
Break rendering...

------

The thing which perplexes me most is this: "Picture viewer uses 'Render' subdivisions. But Live Viewer uses 'Editor' subdivision values.
Try to decrease 'render/editor' subdivison values."
In my scene I only have one subdivision surface applied to an object. It's a cloth subdivision and the subdivisions on this is set to 1.
I really rather need that cloth subdivision on this object...

[abstrax]

aoktar, I may have you on the hook for a potential second drink/lunch...

I began rendering a new clip from the same animation project only to encounter the dreaded (and confounding) CUDA error 700.
So, I tried rendering the scene from where it stopped short and, yet again, I encountered a render failure

Here is an excerpt from the log:

(Frame 226 and 227 rendered as expected, but then, at frame 228, the error is encountered)

FRAME 226 fps:30
MB:0/0 ST/MOV:0/14 Nodes:52 Tris:114k DispTris:0 Hairs:0 Meshes:15
VRAM used/free/max:362Mb/6.076Gb/8Gb Out-of-core used:0Kb total used RAM:7.375Gb
MotBlurTM=0 sec. createTM=0.036 sec. matTM=0 sec. updateTM=0.564 sec.
Device:0 TotMem:8Gb rtData:284Mb film:10Mb geo:24Mb node:9Kb tex:42Mb unavailable:1.357Gb temperature:54
Device:1 TotMem:8Gb rtData:284Mb film:10Mb geo:24Mb node:9Kb tex:42Mb unavailable:1.357Gb temperature:66
Device:2 TotMem:8Gb rtData:284Mb film:10Mb geo:24Mb node:9Kb tex:42Mb unavailable:1.57Gb temperature:60
renderTM:115.805ms. totalTM:116.405ms.


FRAME 227 fps:30
MB:0/0 ST/MOV:0/14 Nodes:52 Tris:115k DispTris:0 Hairs:0 Meshes:15
VRAM used/free/max:362Mb/6.076Gb/8Gb Out-of-core used:0Kb total used RAM:7.4Gb
MotBlurTM=0 sec. createTM=0.04 sec. matTM=0 sec. updateTM=0.528 sec.
Device:0 TotMem:8Gb rtData:284Mb film:10Mb geo:25Mb node:9Kb tex:42Mb unavailable:1.357Gb temperature:52
Device:1 TotMem:8Gb rtData:284Mb film:10Mb geo:25Mb node:9Kb tex:42Mb unavailable:1.357Gb temperature:65
Device:2 TotMem:8Gb rtData:284Mb film:10Mb geo:25Mb node:9Kb tex:42Mb unavailable:1.57Gb temperature:60
renderTM:116.052ms. totalTM:116.62ms.


FRAME 228 fps:30
MB:0/0 ST/MOV:0/14 Nodes:52 Tris:114k DispTris:0 Hairs:0 Meshes:15
VRAM used/free/max:362Mb/6.076Gb/8Gb Out-of-core used:0Kb total used RAM:7.397Gb
MotBlurTM=0 sec. createTM=0.036 sec. matTM=0 sec. updateTM=0.539 sec.
Device:0 TotMem:8Gb rtData:284Mb film:10Mb geo:24Mb node:9Kb tex:42Mb unavailable:1.357Gb temperature:52
Device:1 TotMem:8Gb rtData:284Mb film:10Mb geo:24Mb node:9Kb tex:42Mb unavailable:1.357Gb temperature:65
Device:2 TotMem:8Gb rtData:284Mb film:10Mb geo:24Mb node:9Kb tex:42Mb unavailable:1.57Gb temperature:60
CUDA error 700 on device 1: an illegal memory access was encountered
-> kernel execution failed(report)
CUDA error 700 on device 1: an illegal memory access was encountered
-> failed to copy memory from device.
device 1: path tracing kernel failed
device 1: detected an error on render device! trying to recover...

Is this reproducible? If yes, could you send me the scene please. I will then have a look at what is going on. Thank you.