This is potentially due to irregular geometry in the scene (ie. A poly with only 2 vertices). If you check the octaneplugin.log file you can see which geometry piece caused the issue. The next release will handle irregular geometry in a better way.
If the cuda error is due to some non-standard geometry, then a new card is unlikely to help. If you post your octaneplugin.log file here, and I tell you what geometry is causing the issue.