Hello, this happened in RC2 but also now in RC3 too:
(1) in Standalone I had a scene where I was adding Material layers to a same single Universal material, and at 7 Material layers my GPUs would go red (crash needing restart of Octane).
And I could repeat this - at 6 good, 7 bad.
And no matter how I added then 7th - within a Layer group, as a single layer, with or without added Material Cache in Preferences - it always caused the red-GPU-crash.
But, then to test if this was caused by # of layers I created a very primitive scene with a primitive RGB-color Universal Material and was able to add 9-10-11 Material layers to that same-single Universal material, with no problem.
So I am thinking it was not the # of layers itself, rather something more generic that must be causing the red GPU-crash.
Would it be memory load on a single material? Or basically, what part of a scene - specific to a single material instance- is related to GPUs going Red requiring Octane restart. (ie cumulative Memory load, # textures, # nodes, etc...)
Also an idea...
(2) What was scary was, for this same 7 Material layer scene, I had a Baked Image Texture in, which loads at Scene build.
Now, the first time the 7-Material Layer Red-GPU crash happened, I saved a version of the scene, with 7 Layers, in an attempt to next reload and retry again.
But, I found this saved-7-Material layer scene could no longer load because when the load processes, it would do baking of the baked image texture, and the GPUs would go red (I am thinking again for the 7 Material layers, which had caused the red GPU crash before). This red-GPU on load halted the completion of the load. So I could no longer get into the scene at all on the Graph node.
Luckily, I had also saved a 6 layer one and thus I could continue to work around it, no problem.
But if hadn't had this, I would have lost the scene.
So, my idea
perhaps an emergency rescue 'do not build baking textures on load' could be added somewhere in preferences, to rescue a scene with a sort of Octane 'Safe Mode'?
I don't know how often this might happen, but could be a nice redundancy in case a scene goes crash and won't load anymore.
Win 10 Pro 64, Xeon E5-2687W v2 (8x 3.40GHz), G.Skill 64 GB DDR3-2400, ASRock X79 Extreme 11
Mobo: 1 Titan RTX, 1 Titan Xp
External: 6 Titan X Pascal, 2 GTX Titan X
Plugs: Enterprise