Bug - Network rendering - lost samples when slaves quit Tweet

[abstrax]

same here i noticed large scenes are the worst one of my slaves constantly does it and starting 2.13 octane eats all my ram on main pc during overnight rendering used all 64 gb last night strange

Looks like a memory leak. Could you please provide us a scene that leaks so we can investigate? Is the slave running out of memory or the master or both? Thanks.

[coilbook]

same here i noticed large scenes are the worst one of my slaves constantly does it and starting 2.13 octane eats all my ram on main pc during overnight rendering used all 64 gb last night strange

Looks like a memory leak. Could you please provide us a scene that leaks so we can investigate? Is the slave running out of memory or the master or both? Thanks.

just the master did it. Unfortunately scene is over 1 gb in size with tons of texture and proxies etc but if slave crash i will post crash log here

[Karba]

same here i noticed large scenes are the worst one of my slaves constantly does it and starting 2.13 octane eats all my ram on main pc during overnight rendering used all 64 gb last night strange

Looks like a memory leak. Could you please provide us a scene that leaks so we can investigate? Is the slave running out of memory or the master or both? Thanks.

just the master did it. Unfortunately scene is over 1 gb in size with tons of texture and proxies etc but if slave crash i will post crash log here

You just told "one of my slaves"
So not a slave but master eats all ram?

[coilbook]

You just told "one of my slaves"
So not a slave but master eats all ram?[/quote]
Sorry master one ate all my memory 64 gb and slave had lost samples. But I jsut found out that slave card had bad fan. I will replace the card soon and post a log if still have problems

[coilbook]

Ok, so another version of Octane is out, is there any timeline for when this issue may be fixed. It to me that if you can see the image in the octane frame buffer it cant be too hard to make a button to save direct from frame buffer for a workaround.

For multiple reasons I can't explain in detail it's a bit more complicated than this and it requires some major changes in the render framework. We have it on our to-do list, but I can't tell you when we will have this problem solved. All I can tell you is that I want to have it solved as much as you do.

For now, slaves shouldn't crash. If they do, we need to figure out what is causing the crashes. So if you can reproduce the issue, either post here or send me a PM so we can investigate.

Hi Last night it stopped rendering with green bar almost to the end similar with cuda error 700 but this time is lost samples and Invalid sub-sample mode specified

[coilbook]

here is one more on master

CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> kernel execution failed (dl)
device 3: direct lighting failed
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> failed to deallocate device memory
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> could not get memory info
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> failed to deallocate device memory
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> could not get memory info
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> failed to deallocate device memory
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> could not get memory info
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> failed to deallocate device memory
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> could not get memory info
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> failed to deallocate device memory
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> could not get memory info
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> failed to deallocate device memory
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> could not get memory info
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> failed to deallocate device memory
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> could not get memory info
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> failed to deallocate device memory
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> could not get memory info
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> failed to deallocate device memory
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> could not get memory info
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> failed to deallocate device memory
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> could not get memory info
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> failed to deallocate device memory
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> could not get memory info
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> failed to deallocate device memory
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> could not get memory info
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> failed to deallocate device memory
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> could not get memory info
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> failed to deallocate device memory
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> could not get memory info
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> failed to deallocate device memory
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> could not get memory info
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> failed to deallocate device memory
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> could not get memory info
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> failed to deallocate device memory
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> could not get memory info
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> failed to deallocate device memory
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> could not get memory info
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> failed to deallocate device array
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> could not get memory info
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> failed to deallocate device array
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> could not get memory info
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> failed to deallocate device array
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> could not get memory info
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> failed to deallocate device memory
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> could not get memory info
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> failed to deallocate device array
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> could not get memory info
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> failed to deallocate device array
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> could not get memory info
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> failed to deallocate device array
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> could not get memory info
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> failed to deallocate device array
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> could not get memory info
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> failed to deallocate device array
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> could not get memory info
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> failed to deallocate device array
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> could not get memory info
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> failed to deallocate device array
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> could not get memory info
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> failed to deallocate device array
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> could not get memory info
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> failed to deallocate device array
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> could not get memory info
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> failed to deallocate device array
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> could not get memory info
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> failed to deallocate device array
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> could not get memory info
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> failed to deallocate device array
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> could not get memory info
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> failed to deallocate device array
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> could not get memory info
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> failed to deallocate device array
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> could not get memory info
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> failed to deallocate device array
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> could not get memory info
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> failed to deallocate device array
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> could not get memory info
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> failed to deallocate device array
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> could not get memory info
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> failed to deallocate device array
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> could not get memory info
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> failed to deallocate device array
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> could not get memory info
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> failed to deallocate device array
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> could not get memory info
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> failed to deallocate device array
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> could not get memory info
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> failed to deallocate device array
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> could not get memory info
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> failed to deallocate device array
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> could not get memory info
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> failed to deallocate device array
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> could not get memory info
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> failed to deallocate device array
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> could not get memory info
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> failed to deallocate device array
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> could not get memory info
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> failed to deallocate device array
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> could not get memory info
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> failed to deallocate device array
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> could not get memory info
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> failed to deallocate device array
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> could not get memory info
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> failed to deallocate device array
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> could not get memory info
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> failed to deallocate device array
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> could not get memory info
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> failed to deallocate device array
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> could not get memory info
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> failed to deallocate device array
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> could not get memory info
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> failed to deallocate device array
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> could not get memory info
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> failed to deallocate device array
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> could not get memory info
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> failed to deallocate device array
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> could not get memory info
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> failed to deallocate device array
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> could not get memory info
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> failed to deallocate device array
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> could not get memory info
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> failed to deallocate device array
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> could not get memory info
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> failed to deallocate device array
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> could not get memory info
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> failed to deallocate device array
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> could not get memory info
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> failed to deallocate device array
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> could not get memory info
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> failed to deallocate device array
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> could not get memory info
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> failed to deallocate device array
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> could not get memory info
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> failed to deallocate device array
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> could not get memory info
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> failed to deallocate device array
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> could not get memory info
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> failed to deallocate device array
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> could not get memory info
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> failed to deallocate device array
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> could not get memory info
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> failed to deallocate device array
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> could not get memory info
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> failed to deallocate device array
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> could not get memory info
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> failed to deallocate device array
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> could not get memory info
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> failed to deallocate device array
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> could not get memory info
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> failed to deallocate device array
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> could not get memory info
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> failed to deallocate device array
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> could not get memory info
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> failed to deallocate device array
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> could not get memory info
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> failed to deallocate device array
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> could not get memory info
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> failed to deallocate device array
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> could not get memory info
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> failed to deallocate device array
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> could not get memory info
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> failed to deallocate device array
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> could not get memory info
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> failed to deallocate device array
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> could not get memory info
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> failed to deallocate device array
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.
-> could not get memory info
CUDA error 702 on device 3: The device kernel took too long to execute. This can only occur if timeouts are enabled. The context cannot be used anymore and must be destroyed. All existing device memory allocations from this context are invalid and must be reconstructed.

[abstrax]

Hi Coilbook,

The last problem is a limitation: You are very likely using a lot of displacement mapping, which can be really slow (especially since 2.13 - we are working on it). That means that a kernel call may take longer than the times specified in the TdrDelay settings of the registry. The Standalone installer sets this value to 10 (seconds). If the kernel takes longer, the device driver gets "killed" by Windows. The reason why it does that is because a kernel call on a display GPU that never finishes, will cause the whole system to hang.

Unfortunately, when Octane launches a kernel and it takes too long, there is no way to stop it. In other words, we can't do anything about the timeout it. But you can manually change it to a higher timeout via the registry editor:
- Go to HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\GraphicsDrivers
- The key "TdrLevel" should be set to 3.
- The key "TdrDelay" is the delay in seconds. Maybe set it to 20 or 30.

If that is still not solving the problem and you are using displacement mapping in the scene, you may have stumbled over a displacement bug, that Andrey fixed a few days ago (causing the kernel to hang in an infinite loop).

I hope that helps,
Marcus

[coilbook]

i went back 2.11 and still get Invalid sub-sample mode specified (3)
Invalid sub-sample mode specified (3)

and i never had that error message before I thought it started in 2.13

[coilbook]

Hi Coilbook,

The last problem is a limitation: You are very likely using a lot of displacement mapping, which can be really slow (especially since 2.13 - we are working on it). That means that a kernel call may take longer than the times specified in the TdrDelay settings of the registry. The Standalone installer sets this value to 10 (seconds). If the kernel takes longer, the device driver gets "killed" by Windows. The reason why it does that is because a kernel call on a display GPU that never finishes, will cause the whole system to hang.

Unfortunately, when Octane launches a kernel and it takes too long, there is no way to stop it. In other words, we can't do anything about the timeout it. But you can manually change it to a higher timeout via the registry editor:
- Go to HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\GraphicsDrivers
- The key "TdrLevel" should be set to 3.
- The key "TdrDelay" is the delay in seconds. Maybe set it to 20 or 30.

If that is still not solving the problem and you are using displacement mapping in the scene, you may have stumbled over a displacement bug, that Andrey fixed a few days ago (causing the kernel to hang in an infinite loop).

I hope that helps,
Marcus

Thank you Marcus
I just posted another post before I read this one . We do not use any displacement but use thousands of trees and grass using itoo forest and it takes 30 seconds just to load for each frame. We will reduce it next time Thank you for your advice

[Julien]

Hi Marcus,
I've got the same problem:["the device driver gets "killed" by windows"] and the render fails again and again and stops eventually.
My scene is quite heavy but my hardware should do the job easily. The problem still occurs after removing octane proxys and displacement.
So I've changed the "TdrDelay" to 30 and even 60 but nothing changed.

Honestly, this is a very important problem.
I'm thinking of reversing to another rendering program for this scene. It depresses me to do that but I don't see any other way. Do you?

PS: error log attached

1 GTX TITAN X and 2 GTX TITAN Black
Octane Version 2.21.1
GeForce Driver Version 347.88

Sincerely,
Julien

Hi Coilbook,

The last problem is a limitation: You are very likely using a lot of displacement mapping, which can be really slow (especially since 2.13 - we are working on it). That means that a kernel call may take longer than the times specified in the TdrDelay settings of the registry. The Standalone installer sets this value to 10 (seconds). If the kernel takes longer, the device driver gets "killed" by Windows. The reason why it does that is because a kernel call on a display GPU that never finishes, will cause the whole system to hang.

Unfortunately, when Octane launches a kernel and it takes too long, there is no way to stop it. In other words, we can't do anything about the timeout it. But you can manually change it to a higher timeout via the registry editor:
- Go to HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\GraphicsDrivers
- The key "TdrLevel" should be set to 3.
- The key "TdrDelay" is the delay in seconds. Maybe set it to 20 or 30.

If that is still not solving the problem and you are using displacement mapping in the scene, you may have stumbled over a displacement bug, that Andrey fixed a few days ago (causing the kernel to hang in an infinite loop).

I hope that helps,
Marcus