You have 2GB of RAM on your graphics card - which is not enough. Windows will take nearly half that, and Rhino will take a lot for it's OpenGL display, leaving almost no RAM for Octane. If you have a second card, you could use that for your display adapter, which will free up the space Windows and Rhino are using. You can also enable Out Of Core Textures in the OctaneRender for Rhino plugin so texturemaps are stored in CPU RAM rather than GPU RAM. But 2GB is just not enough for anything serious.