Yes, I have seen this cuda error before working with heavy scenes. I know that at least one bug related with this problem has been fixed and will be available in the next build.
Both GPUs have 12 GB of RAM, right? Can you please try to render your scene only with the GPU not used for display? ... if you render with the GPU used for display, some GPU RAM is not going to be available for render because it is used for display, OpenGL and so on.
Anyway, can you send to me the complete log file?
Computer Graphics Software Developer
Octane plugin development and support: LightWave and Houdini