OTOY Forums

Hey,

I'm regularly getting a CUDA error 999 with Octane in C4D and one of my 2x 980tis drops out during render - even on low sampling.
I've had the machine running Octane with no issues for over a year and it's only started happening recently.


Software/Driver versions:
Octane 3.04.5
nVidia drivers, 378.49
Updated to latest Cuda yesterday to see if that would resolve.

Error log attached from Octane below

Does anyone have any advice / tips I could try to find out the cause...

Is it possibly a hardware issue? I have enough wattage in the PSU (1000w)
I was thinking of swapping the card slot to see if that would resolve?

Thanks a lot in advance.

-

Started logging on 11.02.17 21:44:12

OctaneRender 3.04.5 (3040500)

Scene created in plugin version -1
could not pick, because rendering hasn't started yet
--------------------------- EXPORT LOG ---------------------------
Export materials time= 117.132 ms
Collect objects time= 14.851 ms
Mesh creation time = 99.642 ms.
Triangles:0 Disp.triangles:0 Hairs:0 Meshes:0
VRAM used/free/max:349Mb/4.557Gb/6Gb
Out-of-core used:0Kb total used RAM:4.576Gb
Total export Time = 460.811 ms
CUDA error 999 on device 1: unknown error
-> kernel execution failed(report)
CUDA error 999 on device 1: unknown error
-> failed to launch kernel(ptBrdf2)
device 1: path tracing kernel failed
device 1: detected an error on render device! trying to recover...
CUDA error 999 on device 1: unknown error
-> failed to deallocate device memory
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device memory
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device memory
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device memory
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device memory
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device memory
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device memory
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device memory
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device memory
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device memory
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device memory
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device memory
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device memory
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device memory
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device memory
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device memory
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device memory
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device memory
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device memory
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device memory
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device memory
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device array
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device memory
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate pinned memory
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate pinned memory
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device memory
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate pinned memory
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate pinned memory
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device memory
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device memory
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device memory
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device memory
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device memory
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device memory
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device memory
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device memory
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device memory
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device memory
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device memory
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device memory
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device memory
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to unload module
CUDA error 999 on device 0: unknown error
-> failed to launch kernel(ptBrdf2)
device 0: path tracing kernel failed
device 0: detected an error on render device! trying to recover...
CUDA error 999 on device 0: unknown error
-> failed to deallocate device memory
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device memory
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device memory
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device memory
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device memory
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device memory
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device memory
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device memory
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device memory
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device memory
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device memory
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device memory
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device memory
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device memory
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device memory
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device memory
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device memory
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device memory
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device memory
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device memory
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device array
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device array
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device array
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device memory
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device array
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device memory
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate pinned memory
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate pinned memory
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to unregister pinned memory
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to unregister pinned memory
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device memory
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device memory
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device memory
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device memory
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate pinned memory
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate pinned memory
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device memory
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device memory
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device memory
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device memory
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device memory
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device memory
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device memory
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device memory
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device memory
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device memory
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device memory
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device memory
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device memory
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to unload module
CUDA error 999 on device 1: unknown error
-> failed to bind device to current thread
device 1: failed to initialize resources
device 1: failed to recover - giving up and switching device to failed
device 1: data texture 4 was not initialized
CUDA error 999 on device 0: unknown error
-> failed to bind device to current thread
device 0: failed to initialize resources
device 0: failed to recover - giving up and switching device to failed
----------------------------------------------------------------------------------------
<<< Render failure detected!!! >>>
Please check render statistics to solve the problem.
Triangles:341k Disp.triangles:0 Hairs:0 Meshes:246
Device:0 TotMem:6Gb rtData:0Kb film:0Kb geo:0Kb node:0Kb tex:0Kb unavailable:1.68Gb temperature:69
Device:1 TotMem:6Gb rtData:0Kb film:0Kb geo:0Kb node:0Kb tex:0Kb unavailable:1.504Gb temperature:69

Hey Brezaling,

I am having the same issue. Turn off your second card in octane settings and see if it is still happening. My 4th graphic card is doing the same thing on my machine. I have a feeling it's a hardware issue.

I'm yet to trouble shoot my computer further as I'm halfway through a job. If I make any findings I'll let you know.

Hey,

I seem to have resolved this (touch wood).

I swapped the cards in their slots then I ran a complete uninstall of the nvidia drivers, restarted then installed the latest drivers clean.

I haven't ran into any cuda errors or crashes the past few days and I've done a few heavy render sequences.

It's probably the drivers that did it but switching the card slots or checking they're secured and all cables are connected is worth a shout.

PC tech support advised me to turn on SLI which i didn't do because it's not advised for octane use - but it got me thinking that maybe the card reset fixed some sort of conflict between the two cards running it once - I had the same thing were I could run it on one card fine but then when two were enbaled in Oct. settings that's when I'd get a crash.

B