CUDA error 719 on device *: unspecified launch failure

[teeshiiddo]

I've been having stability issues for a while. Always with this error "CUDA error 719 on device *: unspecified launch failure".

I always render animations, roughly 30sec per frame, all using animated alembic files. It can happen after 10min or 10hours it very unpredictable. I have 5x GTX 780 Ti + Quadro 4000 (monitor only). The fewer cards I use the more stable it seems, but it's hard to be sure. The problem is I want to be able to leave the machine render over the weekend without it stopping.

Things I have tried:

Additional PSU
Underclocking Core and Memory
Setting Fans to 100%
Eliminating each card one by one to test for hardware issues

I use the C4D plugin but have replicated the same issue in the standalone so I don't believe it's plugin related.

I don't know what else I can try? It's becoming a real issue with many upcoming projects. Thanks you for any insights.



Started logging on 29.12.16 11:46:34

OctaneRender 3.05.1 (3050100)

Scene created in plugin version -1

FRAME 407 fps:30 camMb:0 objMb:0
Triangles:1.06m Disp.triangles:0 Hairs:0 Meshes:5
VRAM used/free/max:712Mb/1.652Gb/3Gb
Out-of-core used:0Kb total used RAM:8.585Gb
MotBlurTM=0 sec. createTM=0.524 sec. evalTM=7.229 sec.
Device:0 TotMem:3Gb rtData:341Mb film:7Mb geo:235Mb node:4Kb tex:128Mb unavailable:668Mb temperature:72
Device:1 TotMem:3Gb rtData:341Mb film:7Mb geo:235Mb node:4Kb tex:128Mb unavailable:598Mb temperature:64
Device:2 TotMem:2Gb rtData:0Kb film:0Kb geo:0Kb node:0Kb tex:0Kb unavailable:0Kb temperature:63
Device:3 TotMem:3Gb rtData:341Mb film:7Mb geo:235Mb node:4Kb tex:128Mb unavailable:598Mb temperature:63
Device:4 TotMem:3Gb rtData:341Mb film:7Mb geo:235Mb node:4Kb tex:128Mb unavailable:598Mb temperature:66
Device:5 TotMem:3Gb rtData:0Kb film:0Kb geo:0Kb node:0Kb tex:0Kb unavailable:0Kb temperature:38
CUDA error 719 on device 3: unspecified launch failure
-> kernel execution failed(report)
CUDA error 719 on device 3: unspecified launch failure
-> failed to launch kernel(ptBrdf2)
device 3: path tracing kernel failed
device 3: detected an error on render device! trying to recover...
CUDA error 719 on device 3: unspecified launch failure
-> failed to deallocate device memory
CUDA error 719 on device 3: unspecified launch failure
-> could not get memory info
CUDA error 719 on device 3: unspecified launch failure
-> failed to deallocate device memory
CUDA error 719 on device 3: unspecified launch failure
-> could not get memory info
CUDA error 719 on device 4: unspecified launch failure
CUDA error 719 on device 3: unspecified launch failure
-> failed to deallocate device memory
-> kernel execution failed(report)
CUDA error 719 on device 3: unspecified launch failure
-> could not get memory info
CUDA error 719 on device 3: unspecified launch failure
-> failed to deallocate device memory
CUDA error 719 on device 4: unspecified launch failure
-> failed to launch kernel(ptBrdf2)
CUDA error 719 on device 3: unspecified launch failure
-> could not get memory info
device 4: path tracing kernel failed
device 4: detected an error on render device! trying to recover...
CUDA error 719 on device 3: unspecified launch failure
-> failed to deallocate device memory
CUDA error 719 on device 4: unspecified launch failure
CUDA error 719 on device 3: unspecified launch failure
-> failed to deallocate device memory
-> could not get memory info
CUDA error 719 on device 4: unspecified launch failure
CUDA error 719 on device 3: unspecified launch failure
-> could not get memory info
-> failed to deallocate device memory
CUDA error 719 on device 3: unspecified launch failure
-> could not get memory info
CUDA error 719 on device 4: unspecified launch failure
-> failed to deallocate device memory
CUDA error 719 on device 3: unspecified launch failure
-> failed to deallocate device memory
CUDA error 719 on device 4: unspecified launch failure
-> could not get memory info
CUDA error 719 on device 3: unspecified launch failure
-> could not get memory info
CUDA error 719 on device 4: unspecified launch failure
CUDA error 719 on device 3: unspecified launch failure
-> failed to deallocate device memory
-> failed to deallocate device memory
CUDA error 719 on device 4: unspecified launch failure
CUDA error 719 on device 3: unspecified launch failure
-> could not get memory info
-> could not get memory info
CUDA error 719 on device 4: unspecified launch failure
-> failed to deallocate device memory
CUDA error 719 on device 4: unspecified launch failure
CUDA error 719 on device 3: unspecified launch failure
-> could not get memory info
-> failed to deallocate device memory
CUDA error 719 on device 3: unspecified launch failure
-> could not get memory info
CUDA error 719 on device 4: unspecified launch failure
-> failed to deallocate device memory
CUDA error 719 on device 3: unspecified launch failure
-> failed to deallocate device memory
CUDA error 719 on device 4: unspecified launch failure
-> could not get memory info
CUDA error 719 on device 3: unspecified launch failure
-> could not get memory info
CUDA error 719 on device 4: unspecified launch failure
-> failed to deallocate device memory
CUDA error 719 on device 3: unspecified launch failure
-> failed to deallocate device memory
CUDA error 719 on device 4: unspecified launch failure
-> could not get memory info
CUDA error 719 on device 3: unspecified launch failure
-> could not get memory info
CUDA error 719 on device 4: unspecified launch failure
-> failed to deallocate device memory
CUDA error 719 on device 3: unspecified launch failure
CUDA error 719 on device 4: unspecified launch failure
-> failed to deallocate device memory
-> could not get memory info
CUDA error 719 on device 3: unspecified launch failure
CUDA error 719 on device 4: unspecified launch failure
-> could not get memory info
-> failed to deallocate device memory
CUDA error 719 on device 3: unspecified launch failure
CUDA error 719 on device 4: unspecified launch failure
-> failed to deallocate device memory
-> could not get memory info
CUDA error 719 on device 3: unspecified launch failure
CUDA error 719 on device 4: unspecified launch failure
-> could not get memory info
-> failed to deallocate device memory
CUDA error 719 on device 3: unspecified launch failure
CUDA error 719 on device 4: unspecified launch failure
-> failed to deallocate device memory
-> could not get memory info
CUDA error 719 on device 3: unspecified launch failure
CUDA error 719 on device 4: unspecified launch failure
-> could not get memory info
-> failed to deallocate device memory
CUDA error 719 on device 3: unspecified launch failure
CUDA error 719 on device 4: unspecified launch failure
-> failed to deallocate device memory
-> could not get memory info
CUDA error 719 on device 3: unspecified launch failure
CUDA error 719 on device 4: unspecified launch failure
-> could not get memory info
-> failed to deallocate device memory
CUDA error 719 on device 3: unspecified launch failure
CUDA error 719 on device 4: unspecified launch failure
-> failed to deallocate device memory
-> could not get memory info
CUDA error 719 on device 3: unspecified launch failure
CUDA error 719 on device 4: unspecified launch failure
-> could not get memory info
-> failed to deallocate device memory
CUDA error 719 on device 3: unspecified launch failure
CUDA error 719 on device 4: unspecified launch failure
-> failed to deallocate device memory
-> could not get memory info
CUDA error 719 on device 3: unspecified launch failure
CUDA error 719 on device 4: unspecified launch failure
-> could not get memory info
-> failed to deallocate device memory
CUDA error 719 on device 3: unspecified launch failure
CUDA error 719 on device 4: unspecified launch failure
-> failed to deallocate device memory
-> could not get memory info
CUDA error 719 on device 3: unspecified launch failure
CUDA error 719 on device 4: unspecified launch failure
-> could not get memory info
-> failed to deallocate device memory
CUDA error 719 on device 3: unspecified launch failure
CUDA error 719 on device 4: unspecified launch failure
-> failed to deallocate device memory
-> could not get memory info
CUDA error 719 on device 3: unspecified launch failure
CUDA error 719 on device 4: unspecified launch failure
-> could not get memory info
-> failed to deallocate device memory
CUDA error 719 on device 3: unspecified launch failure
CUDA error 719 on device 4: unspecified launch failure
-> failed to deallocate device memory
-> could not get memory info
CUDA error 719 on device 3: unspecified launch failure
CUDA error 719 on device 4: unspecified launch failure
-> could not get memory info
-> failed to deallocate device memory
CUDA error 719 on device 4: unspecified launch failure
-> could not get memory info
CUDA error 719 on device 3: unspecified launch failure
-> failed to deallocate device array
CUDA error 719 on device 4: unspecified launch failure
-> failed to deallocate device memory
CUDA error 719 on device 3: unspecified launch failure
-> could not get memory info
CUDA error 719 on device 4: unspecified launch failure
-> could not get memory info
CUDA error 719 on device 3: unspecified launch failure
-> failed to deallocate device array
CUDA error 719 on device 4: unspecified launch failure
-> failed to deallocate device memory
CUDA error 719 on device 3: unspecified launch failure
-> could not get memory info
CUDA error 719 on device 4: unspecified launch failure
-> could not get memory info
CUDA error 719 on device 3: unspecified launch failure
-> failed to deallocate device array
CUDA error 719 on device 4: unspecified launch failure
-> failed to deallocate device memory
CUDA error 719 on device 3: unspecified launch failure
-> could not get memory info
CUDA error 719 on device 4: unspecified launch failure
-> could not get memory info
CUDA error 719 on device 3: unspecified launch failure
-> failed to deallocate device memory
CUDA error 719 on device 4: unspecified launch failure
-> failed to deallocate device memory
CUDA error 719 on device 3: unspecified launch failure
-> could not get memory info
CUDA error 719 on device 4: unspecified launch failure
-> could not get memory info
CUDA error 719 on device 3: unspecified launch failure
CUDA error 719 on device 4: unspecified launch failure
-> failed to deallocate device array
-> failed to deallocate device array
CUDA error 719 on device 3: unspecified launch failure
CUDA error 719 on device 4: unspecified launch failure
-> could not get memory info
-> could not get memory info
CUDA error 719 on device 3: unspecified launch failure
CUDA error 719 on device 4: unspecified launch failure
-> failed to deallocate device array
-> failed to deallocate device array
CUDA error 719 on device 3: unspecified launch failure
CUDA error 719 on device 4: unspecified launch failure
-> could not get memory info
-> could not get memory info
CUDA error 719 on device 4: unspecified launch failure
CUDA error 719 on device 3: unspecified launch failure
-> failed to deallocate device array
-> failed to deallocate device memory
CUDA error 719 on device 4: unspecified launch failure
CUDA error 719 on device 3: unspecified launch failure
-> could not get memory info
-> could not get memory info
CUDA error 719 on device 4: unspecified launch failure
-> failed to deallocate device memory
CUDA error 719 on device 3: unspecified launch failure
-> failed to deallocate pinned memory
CUDA error 719 on device 4: unspecified launch failure
-> could not get memory info
CUDA error 719 on device 3: unspecified launch failure
-> could not get memory info
CUDA error 719 on device 4: unspecified launch failure
CUDA error 719 on device 3: unspecified launch failure
-> failed to deallocate device array
-> failed to deallocate pinned memory
CUDA error 719 on device 4: unspecified launch failure
CUDA error 719 on device 3: unspecified launch failure
-> could not get memory info
-> could not get memory info
CUDA error 719 on device 4: unspecified launch failure
-> failed to deallocate device array
CUDA error 719 on device 4: unspecified launch failure
-> could not get memory info
CUDA error 719 on device 4: unspecified launch failure
-> failed to deallocate device memory
CUDA error 719 on device 4: unspecified launch failure
-> could not get memory info
CUDA error 719 on device 3: unspecified launch failure
-> could not get memory info
CUDA error 719 on device 4: unspecified launch failure
-> failed to deallocate pinned memory
CUDA error 719 on device 3: unspecified launch failure
-> could not get memory info
CUDA error 719 on device 4: unspecified launch failure
-> could not get memory info
CUDA error 719 on device 3: unspecified launch failure
CUDA error 719 on device 4: unspecified launch failure
-> failed to deallocate device memory
-> failed to deallocate pinned memory

[coilbook]

Maybe cards are getting too hot, low ram or gpu memory or scattering very high poly mesh

[smicha]

Are you using risers for 6 gpus overall? What is your motherboard and PSU?

[aoktar]

Try to test your gpus with one gpu. You can't work stable if it's about gpu or dependencies

[teeshiiddo]

Try to test your gpus with one gpu. You can't work stable if it's about gpu or dependencies

I have tried eliminating each card one by one to test for hardware issues, without luck.

When I use 1 or 2 GPUs it is a lot more stable but still happens, just less frequently. But obviously the renders are much slower.

Do you think it could be a hardware issue?

It has been consistent over all my driver updates, since I purchased the equipment in May 2016.

[teeshiiddo]

Are you using risers for 6 gpus overall? What is your motherboard and PSU?

The 5x 780ti are in a JMR Silverstor DT4U-EXTN ( 5x Dual Height 8x PCIe expansion unit )

The Quadro 4000 is in a HP Z800

PSU -
Z800 has an 850W internal
JMR Silverstor has Evga G2 1300 ( but I also tried running 2x of the GPU off a separate 1200W PSU made no difference)

[teeshiiddo]

Maybe cards are getting too hot, low ram or gpu memory or scattering very high poly mesh

I think from the scene stats in the octane log it doesn't seem to be an issue with any of the things.

I have been keeping a close eye on temps. GPU-Z shows no thermal throttling.

[teeshiiddo]

Just to follow up.

I haven't found a solution to the underlying issue, but now using Deadline render manager to navigate the issue.

Breaking the renders into 5 frame chucks the manager calculates and maximum duration for that render chuck, once this time expires it restarts C4D command line and that chuck. Fixing the Cuda error temporarily, and allowing the render to continue. It's been running all weekend, with a few errors but not hanging and not missing frames

Hopefully this helps someone else with similar stability issues.

[zoppo]

I have the same issue from time to time, definitely not a hardware problem.

It only happened after I had updated from 3.0.4.5 to 3.0.5.1

[abstrax]

I have the same issue from time to time, definitely not a hardware problem.

It only happened after I had updated from 3.0.4.5 to 3.0.5.1

Could you try downgrading to 3.04.5 again and test again? If the problem goes away, could you then upgrade to 3.05.x again and check if the problem comes back? If it does, could you send us the scene plus some instructions how to break CUDA?