the issue with iMac is that the internal GPU is already stressed by the big screen resolution leaving only few VRAM for rendering.
The only solution is to add external GPU via Thunderbolt. It's a bit tricky because is needed to modify some kext system files, but it works. There are some users here on the forum adopting this solution (with trash can dyson Mac Pro 2013 also), here you can find all the necessary info:viewtopic.php?f=9&t=43439
Since Apple is always a bit different, the easiest option for working with Octane on Mac is still the old Mac Pro 2009/2012 because is almost plug and play