to use OctaneRender under High Sierra, you need an Nvidia GPU,
Have a look at this alternative eGPU solution via Thunderbolt https://render.otoy.com/forum/viewtopic.php?f=9&t=43439&start=200#p280842
In this way, you can use the internal GPU for system/monitor, while the external GPU is used only for rendering, at full power and with all VRAM.
High Sierra is supporting eGPU solutions by default, without external script.
you need to update to 10.13.2 , then update the Mac CUDA drivers to the following version:
CUDA 387.99 http://www.nvidia.com/object/macosx-cuda-387.99-driver.htm
Update Nvidia GPU graphics drivers http://www.nvidia.com/download/driverResults.aspx/127670/en-us
Then your Nvidia GPU should be correctly recognized, and Octane should be able to start.