The Quadro 5600 is quite an old card based on a GeForce 8800 Ultra. Its general level of performance for gpu rendering would not be very good having 128 cuda cores. It also has cuda Compute Capability 1.0. It may not be so suitable for recent Octane builds but the Otoy guys can advise you about that. Which Nvidia driver are you using at the moment? How many polys has your model? Try closing some open apps and turning off Aero to free some vram
I am pretty sure that Octane 1.02 will only work with GTX 500 and 600 series cards at the moment, so for the moment you should use the 1.00p build of the plugin. If you are running out of memory check the FAQ thread in this forum for tips, including using the texture file compression script supplied with the plugin.
i going buy KFA2 GeForce GTX 680 EX OC 4096MB GDDR5 PCI-Express Graphics Card
i take it one these would work quite well?
I have the GTX680 4GB and am extrem ly happy about it Keep the old card in as well. Plug your monitor in that one and the OS UI will use that card freeing up more memory for the GTX and a much more responsive UI
Configuration: Windows 10 64bit Creators Edition, I7 6850K, 128GB, 2 GeForce 1080ti, P11 Pro SR6