This creates problems when running overnight, as you can have a frame freeze at 2am and only find out in the morning that the time was wasted.
The slaves are a mix of 1080ti and 980ti in pairs (8 slaves, 16 cards total), we have tested as much as we can and cannot find a specific trigger. It's not memory, sometimes a 1080ti will fail when the 980ti are fine.
It seems to happen most often when a slave joins a job part way through, but sometimes a slave will fail that has been working since the beginning of the job and cause the same problem.
Removing and re-adding the slave works, so there's no consistency there either. All slaves are running the same drivers: 382.33 and Octane 3.06 stable.
- Job paused on 5th frame. For 14 hours.
- 2017-11-10 08_46_52-Picture Viewer.png (14.88 KiB) Viewed 2108 times
As you can see, a slave crashed on the 5th frame, and the frame paused at 99.97% for 14 hours. When the slave was restarted everything ran fine again.
- Slave failed 5 frames in. When restarted it worked fine and completed the job
In this case, the error was CUDA 719, but this is not consistent - it sometimes be an error of "not receiving all information for a frame", or sometimes simply say the "slave crashed or was stopped (CTRL-C)"
I understand there are all sorts of variables that could affect rendering so I'm not looking for a specific fix, but rather:
Is there any way to automatically force a slave daemon to quit if it fails - so that it is removed from the pool and the frame will finish?
I'd rather that the job continued a little slower overnight than froze for hours at a time.
Thanks in advance,
James
