Hi,
I have big issues with the new stable version (3.08.09). it is crashing with a scene working well in 3.07.
Dialog box tell me there is a gpu Failure and stop the render. I don't understand why.
my driver version is the latest 398.11. i tried to unactive card by card ( i have only two one )..nothing is working..
Thx to help me !.
Here is the octane log:
Started logging on 26.06.18 11:43:08
OctaneRender 3.08.1 (3080100)
CUDA error 2 on device 0: out of memory
-> failed to create context for pinned memory
device 0: failed to pin film buffer
CUDA error 999 on device 0: unknown error
-> failed to deallocate device array
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device array
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device array
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device array
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device array
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device array
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device array
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device array
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device array
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device array
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device array
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device array
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device array
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device array
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device array
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device array
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device array
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device array
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device array
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device array
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device array
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device array
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device array
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device array
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device array
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device array
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device array
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device array
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device array
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device array
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device array
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device array
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device array
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device array
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device array
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device array
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device array
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device array
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device array
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device array
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device array
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device array
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device array
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device array
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device array
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device array
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device array
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device array
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device array
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device array
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device array
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device array
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device array
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device array
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device array
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device array
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device array
CUDA error 999 on device 0: unknown error
-> failed to deallocate device array
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device array
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device array
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device array
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device array
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device array
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device array
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device array
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device array
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device array
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device array
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device array
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device array
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device array
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device array
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device array
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device array
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device array
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device array
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device array
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device array
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device array
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device array
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device array
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device array
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device array
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device array
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device array
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device array
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device array
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device array
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device array
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device array
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device array
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device array
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device array
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device array
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device array
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device array
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device array
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device array
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device array
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device array
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device array
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device array
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device array
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device array
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device array
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device array
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device array
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device array
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device array
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device array
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device array
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device array
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device array
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device array
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device memory
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate pinned memory
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate pinned memory
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device memory
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate pinned memory
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate pinned memory
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device memory
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device memory
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device memory
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device memory
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device memory
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device memory
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device memory
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device memory
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device memory
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device memory
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device memory
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device memory
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device memory
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device memory
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to deallocate device memory
CUDA error 999 on device 0: unknown error
-> could not get memory info
CUDA error 999 on device 0: unknown error
-> failed to unload module
CUDA error 999 on device 0: unknown error
-> failed to unload module
CUDA error 999 on device 1: unknown error
-> failed to deallocate device memory
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device memory
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device memory
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device memory
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device memory
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device memory
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device memory
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device memory
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device memory
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device memory
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device memory
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device memory
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device memory
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device memory
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device memory
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device memory
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device memory
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device memory
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device memory
CUDA error 999 on device 1: unknown error
-> could not get memory info
Failed to free page-locked memory
Failed to free page-locked memory
Failed to free page-locked memory
Failed to free page-locked memory
Failed to free page-locked memory
Failed to free page-locked memory
Failed to free page-locked memory
CUDA error 999 on device 1: unknown error
-> failed to deallocate device array
CUDA error 999 on device 1: unknown error
-> could not get memory info
Failed to free page-locked memory
Failed to free page-locked memory
Failed to free page-locked memory
CUDA error 999 on device 1: unknown error
-> failed to deallocate device array
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device array
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device array
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device array
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device array
CUDA error 999 on device 1: unknown error
-> could not get memory info
Failed to free page-locked memory
Failed to free page-locked memory
Failed to free page-locked memory
Failed to free page-locked memory
CUDA error 999 on device 1: unknown error
-> failed to deallocate device array
CUDA error 999 on device 1: unknown error
-> could not get memory info
Failed to free page-locked memory
Failed to free page-locked memory
Failed to free page-locked memory
Failed to free page-locked memory
Failed to free page-locked memory
Failed to free page-locked memory
Failed to free page-locked memory
Failed to free page-locked memory
Failed to free page-locked memory
Failed to free page-locked memory
Failed to free page-locked memory
Failed to free page-locked memory
CUDA error 999 on device 1: unknown error
-> failed to deallocate device array
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device array
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device array
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device array
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device array
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device array
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device array
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device array
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device array
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device array
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device array
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device array
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device array
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device array
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device array
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device array
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device array
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device array
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device array
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device array
CUDA error 999 on device 1: unknown error
-> could not get memory info
Failed to free page-locked memory
Failed to free page-locked memory
Failed to free page-locked memory
Failed to free page-locked memory
Failed to free page-locked memory
Failed to free page-locked memory
Failed to free page-locked memory
Failed to free page-locked memory
Failed to free page-locked memory
Failed to free page-locked memory
Failed to free page-locked memory
Failed to free page-locked memory
CUDA error 999 on device 1: unknown error
-> failed to deallocate device array
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device array
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device array
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device array
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device array
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device array
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device array
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device array
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device array
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device array
CUDA error 999 on device 1: unknown error
-> could not get memory info
Failed to free page-locked memory
Failed to free page-locked memory
Failed to free page-locked memory
Failed to free page-locked memory
Failed to free page-locked memory
CUDA error 999 on device 1: unknown error
-> failed to deallocate device array
CUDA error 999 on device 1: unknown error
-> could not get memory info
Failed to free page-locked memory
CUDA error 999 on device 1: unknown error
-> failed to deallocate device array
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device array
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device array
CUDA error 999 on device 1: unknown error
-> could not get memory info
Failed to free page-locked memory
Failed to free page-locked memory
Failed to free page-locked memory
Failed to free page-locked memory
Failed to free page-locked memory
Failed to free page-locked memory
Failed to free page-locked memory
Failed to free page-locked memory
Failed to free page-locked memory
Failed to free page-locked memory
Failed to free page-locked memory
Failed to free page-locked memory
Failed to free page-locked memory
Failed to free page-locked memory
Failed to free page-locked memory
Failed to free page-locked memory
Failed to free page-locked memory
Failed to free page-locked memory
Failed to free page-locked memory
Failed to free page-locked memory
CUDA error 999 on device 1: unknown error
-> failed to deallocate device array
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device array
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device array
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device array
CUDA error 999 on device 1: unknown error
-> could not get memory info
Failed to free page-locked memory
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device array
CUDA error 999 on device 1: unknown error
-> could not get memory info
Failed to free page-locked memory
CUDA error 999 on device 1: unknown error
-> failed to deallocate device array
CUDA error 999 on device 1: unknown error
-> could not get memory info
Failed to free page-locked memory
CUDA error 999 on device 1: unknown error
-> failed to deallocate device array
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device array
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device array
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device array
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device array
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device array
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device array
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device array
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device array
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device array
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device array
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device array
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device array
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device array
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device array
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device array
CUDA error 999 on device 1: unknown error
-> could not get memory info
Failed to free page-locked memory
CUDA error 999 on device 1: unknown error
-> failed to deallocate device array
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device array
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device array
CUDA error 999 on device 1: unknown error
-> could not get memory info
Failed to free page-locked memory
CUDA error 999 on device 1: unknown error
-> failed to deallocate device array
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device array
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device array
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device array
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device array
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device array
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device array
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device array
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device array
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device array
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device array
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device array
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device array
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device array
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device array
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device array
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device array
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device array
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device array
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device array
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device array
CUDA error 999 on device 1: unknown error
-> could not get memory info
Failed to free page-locked memory
CUDA error 999 on device 1: unknown error
-> failed to deallocate device array
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device array
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device array
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device array
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device array
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device array
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device array
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device array
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device array
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device array
CUDA error 999 on device 1: unknown error
-> could not get memory info
Failed to free page-locked memory
CUDA error 999 on device 1: unknown error
-> failed to deallocate device array
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device array
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device array
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device array
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device memory
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate pinned memory
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate pinned memory
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device memory
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate pinned memory
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate pinned memory
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device memory
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device memory
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device memory
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device memory
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device memory
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device memory
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device memory
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device memory
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device memory
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device memory
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device memory
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device memory
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device memory
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device memory
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to deallocate device memory
CUDA error 999 on device 1: unknown error
-> could not get memory info
CUDA error 999 on device 1: unknown error
-> failed to unload module
CUDA error 999 on device 1: unknown error
-> failed to unload module