Sorry, I don't used any other button than I described, so I don't understand what you are missing. I pressed three buttons per mouse, what should I write more?
I did some more test and found, the Octane UI needs to be opened before. So, maybe this steps help:
* Start Rhino
* open the Octane panel (my panel at the right is open and I need to press the Octane symbol to show the options only)
* open new file per Rhino menu
* -> error
I am still not getting this error. Perhaps your default rendertarget settings are corrupt? Try deleting C:\Users\[username]\AppData\Local\OctaneRender for Rhinoceros\Defaults\rendertarget.xml.
Also, which version of Rhino and the plugin are you using please?
Thanks
Paul