Alright, there has been some progress. I pushed the out of core memory to 12gb, I was able to render 9 frames but after that the same error. I have a feeling that the amount of frames I can render is proportional to the out of core memory I assign, maybe there is something which is not properly cleared from the memory between frames and eventually it overflows?
Another thing, it won't let me allocate more than about 6gb memory until I open the viewport, but that is minor issue. This is how the system window looks like when the scene loads, there should be plenty memory to go around:
http://imgur.com/a/cJvzj
I will try to push ooc memory as high as I can and see how many I can render.
Here are the error messages I get:
17:06:28.761: +++++ Plugin running.
18:29:17.617: CUDA error 2 on device 1: The API was unable to allocate enough memory to perform the requested operation.
18:29:17.618: -> failed to deallocate device array
18:29:17.619: CUDA error 2 on device 1: The API was unable to allocate enough memory to perform the requested operation.
18:29:17.619: -> could not get memory info
18:29:17.620: CUDA error 2 on device 0: The API was unable to allocate enough memory to perform the requested operation.
18:29:17.622: -> could not get memory info
18:29:17.623: CUDA error 2 on device 1: The API was unable to allocate enough memory to perform the requested operation.
18:29:17.624: Failed to allocate page-locked memory
18:29:17.625: CUDA error 2 on device 0: The API was unable to allocate enough memory to perform the requested operation.
18:29:17.626: -> failed to copy memory to device.
18:29:17.628: CUDA error 2 on device 2: The API was unable to allocate enough memory to perform the requested operation.
18:29:17.628: -> failed to deallocate device memory
18:29:17.629: CUDA error 2 on device 1: The API was unable to allocate enough memory to perform the requested operation.
18:29:17.629: -> could not get memory info
18:29:17.630: device 1: failed to upload data texture 28 of context 0
18:29:17.636: -> failed to deallocate device memory
18:29:17.636: CUDA error 2 on device 2: The API was unable to allocate enough memory to perform the requested operation.
18:29:17.638: -> could not get memory info
18:29:17.638: device 2: failed to upload data texture 28 of context 0
18:29:17.639: Render engine failure occured! (WARNING)
18:29:17.639: device 0: failed to upload data texture 28 of context 0
18:29:17.640: CUDA error 2 on device 0: The API was unable to allocate enough memory to perform the requested operation.
18:29:17.640: -> failed to deallocate device memory
18:29:17.641: CUDA error 2 on device 0: The API was unable to allocate enough memory to perform the requested operation.
18:29:17.641: -> could not get memory info
18:29:17.642: CUDA error 2 on device 0: The API was unable to allocate enough memory to perform the requested operation.
18:29:17.642: -> failed to copy memory to device.
18:29:17.643: device 0: failed to upload data texture 22 of context 0
18:29:17.643: CUDA error 2 on device 0: The API was unable to allocate enough memory to perform the requested operation.
18:29:17.644: -> failed to deallocate device memory
18:29:17.644: CUDA error 2 on device 0: The API was unable to allocate enough memory to perform the requested operation.
18:29:17.645: -> could not get memory info
18:29:20.635: Render engine failure occured! (WARNING)
18:29:23.634: Render engine failure occured! (WARNING)