Bug - Network rendering - lost samples when slaves quit

Wed Oct 22, 2014 12:26 am

Hi Karba,

Bug Network rendering.
OctaneRender® for 3ds max® v2.11.1a
Windows 7 64bit

Octane loses samples when copying to max render buffer. (We just lost 7 gpu's worth of samples over 8 hours. makes me sad. //12000 samples gone)
Some slaves remained connected and their samples were ok. Some slaves and their samples can drop out & fail during the copy to max frame buffer. Octane fumbles the sample count and has no memory of the samples. This has been a recurrent theme on complex scenes, reasonably high resolution and working with network rendering since its implementation.

How to avoid this?
It appears that the master computer/s struggle with the copy to max frame buffer and cannot multi-task during this period. There also needs to be an underlying strategy for a graceful disconnection of slaves so that they never take their samples with them when they quit.

Also, at some time can we implement saving directly from octane buffer?

Regards,
N.

Wed Oct 22, 2014 2:46 am

This is a real pain, it makes distributed rendering almost pointless, and wastes huge amount of computer render times. Also we have been having this bug for quite a while. it really needs to be on the highest priority on OTOY devlist. Maybe there needs to be the ability to save from the Octane buffer as its not working sending to the Max frame buffer.

Wed Oct 22, 2014 3:17 am

There is nothing I can do here. All slaves keep result on it's side, and if something happens there it will be lost.
Marcus is working on it.

Wed Oct 22, 2014 6:03 am

Hi Karba, thanks for the reply. In Corona the cached samples are all saved to drive. Couldnt Octane progressively save each sample set incase of crashes. Or have Octane autosave the image to disk at different increments eg 2000,3000,4000 samples.

Another option would be saving direct from the Octane frame buffer, why isnt that possible?

Tue Oct 28, 2014 3:25 am

Any more thoughts on this?

Tue Oct 28, 2014 3:30 am

3dgeeks wrote:Any more thoughts on this?

We understand the issue, but the problem is that not everything is done only on the GPU. Otherwise you multi-GPU rendering wouldn't scale well with many GPUs. I think I have a way how to solve it, but didn't have time to work on it yet. Since this problem moved up in our to-do list, it will hopefully be tackled soon.

Mon Nov 03, 2014 6:26 am

Yes it would be good to get some of the bugs ironed out soon, as it makes it a waste of expensive hardware ,and Octane licenses that are not contributing to our output.

Mon Dec 08, 2014 9:06 am

Ok, so another version of Octane is out, is there any timeline for when this issue may be fixed. It to me that if you can see the image in the octane frame buffer it cant be too hard to make a button to save direct from frame buffer for a workaround.

Mon Dec 08, 2014 8:59 pm

same here i noticed large scenes are the worst one of my slaves constantly does it and starting 2.13 octane eats all my ram on main pc during overnight rendering used all 64 gb last night strange

Mon Dec 08, 2014 9:12 pm

3dgeeks wrote:Ok, so another version of Octane is out, is there any timeline for when this issue may be fixed. It to me that if you can see the image in the octane frame buffer it cant be too hard to make a button to save direct from frame buffer for a workaround.

For multiple reasons I can't explain in detail it's a bit more complicated than this and it requires some major changes in the render framework. We have it on our to-do list, but I can't tell you when we will have this problem solved. All I can tell you is that I want to have it solved as much as you do.

For now, slaves shouldn't crash. If they do, we need to figure out what is causing the crashes. So if you can reproduce the issue, either post here or send me a PM so we can investigate.