I have tried to use this to do a howie scene as my 770 only has 2GB of vram, I have set it to use about 8GB of memory for textures, I have observed the memory in use by Carrara shoot up at a rate of knots, task manager says nothing is responding but if you leave it it comes back to life, but there is nothing happening in the octane viewer, I know it says using out of core will be slower, but I can believe that it would be a snails pace :p
Is there some special trick to it, or are Howies scene just the straw that will break any card :p
How do you use out of core?
Howie scenes are actually quite light on texture usage so OOC won't help much. The problem is the trees. They use scattering for the leaves and when you replicate the trees you run out of memory because of the number of leaves in the scene. This has been discussed before and you should search through this forum for ideas on how to get around the problem. I will see if a can dig up some more info...
Win8/64, I7-4770K (3.5Ghz) 24GB, GTX-980ti (6GB) / GTX-TITAN (6GB)
How to fit Howie's Country Lane into Octane...
1) Open an empty scene, Click on Scene then Edit in Octane Render Target to open the Octane Render for Carrara Settings window.
2) Select Preferences and turn off High Res Trees.
3) Load up Country Lane
4) You need to flatten the following trees:
a. Trees->Young Birch Trees->Birch Sapling
b. Trees->Replicated Trees->Alder 1, Alder 2, Birch
c. Background Trees -> Background Alders -> Alder Tree
To flatten a tree, select the tree in the Instances list, select the Effects tab and then locate and select the 'flatten' check box under 'Octane Tree'.
Following these guide lines I was able to load Country Lane into 1.47G of GPU memory.
1) Open an empty scene, Click on Scene then Edit in Octane Render Target to open the Octane Render for Carrara Settings window.
2) Select Preferences and turn off High Res Trees.
3) Load up Country Lane
4) You need to flatten the following trees:
a. Trees->Young Birch Trees->Birch Sapling
b. Trees->Replicated Trees->Alder 1, Alder 2, Birch
c. Background Trees -> Background Alders -> Alder Tree
To flatten a tree, select the tree in the Instances list, select the Effects tab and then locate and select the 'flatten' check box under 'Octane Tree'.
Following these guide lines I was able to load Country Lane into 1.47G of GPU memory.
Win8/64, I7-4770K (3.5Ghz) 24GB, GTX-980ti (6GB) / GTX-TITAN (6GB)
Thanks Sighman, will give your advise a try when I have woken up and had a few cups of coffee 
edit: Yip it's the trees, I removed them from the scene and it renders fast in C8.1, I did all you suggested but it didn't seem to help, as you can see from the image.
Edit2: left it while I went and had a shower leaving only the background trees in, seems to be rendering :p - will try a few other things see if I can get all the trees in it
Edit3: On this scene - Countryside seems the fir trees cause issues, I can render with them removed from the scene, with them in I get the issues described, Don't know if that will help

edit: Yip it's the trees, I removed them from the scene and it renders fast in C8.1, I did all you suggested but it didn't seem to help, as you can see from the image.
Edit2: left it while I went and had a shower leaving only the background trees in, seems to be rendering :p - will try a few other things see if I can get all the trees in it
Edit3: On this scene - Countryside seems the fir trees cause issues, I can render with them removed from the scene, with them in I get the issues described, Don't know if that will help