Node crash halts render

[BorisGoreta]

Maybe it could be adaptive, average time of all times calculated after the first frame is rendered and then wait 2x this time the next frame. Then on the next frame calculate again. If node is not finished in this time kill it and continue.

Currently it waits forever, which is not a good starting point.

People usually sit on a couple of first frame to see if everything is alright and then go to sleep.

I am so very disappointed when in the morning I see the render is still on frame 50 out of 2000. Huge waste of time.

I know I have a faulty hardware or a cable somewhere, but with 19 GPUs it is not always easy to maintain it in a perfect shape. I am doing my best, I have 20 extra GPU cables and swap them when there is something wrong.

Also I had 5 GPUs die on me and they don't die immediately but give you headaches like this for weeks and only then decide to completely die so it can be quite tricky identifying what is the root of the problem.

If such heartbeat system existed it would help a lot.