Let me think about it, but is not going to be easy (or even possible) because in Octane the layers work at object level. One object can't handle several masks. You can use the material masks, but of course this is not the same, because two objects with the same material are going to have the same mask.
I will think about it, or fill a feature request to add this feature to Octane.
Computer Graphics Software Developer
Octane plugin development and support: LightWave and Houdini