Network render issues

Forums: Network render issues
Sub forum for help and tutorials.

Moderator: aoktar

Network render issues

Postby aoktar » Tue Jun 23, 2015 9:44 pm

aoktar Tue Jun 23, 2015 9:44 pm
It's quoted from Abstrax's post:

Although network rendering looks simple and usually just "magically" does it's work, it's quite complex inside. So there are many things that can go wrong here. At the moment, if a slave crashes or fails there are usually no good error messages on the master. So you would have to check the console output of the slave process on the slave computer. And we try to make sure that the slave doesn't crash. In version 3 we plan to make things more robust. We are right now working in that area.

Anyway, to give you an idea what happens internally, this is the flow on a high level when an image is rendered via network rendering (assuming the master <-> slave connection is already established):

1) The master sends the current render data (geometry, textures, etc.) to the slave.
2) The master assigns some work to the slave.
3) The slave starts working through the assigned work.
4) After some work is finished, it sends a result back to the master and continues working through the assignment.
5) After receiving the result the master assigns more work to the slave unless all samples are done.
6) The master uses the result of this slave, other slaves and the local GPUs to create the current render result, which is passed on to the plugin.
7) go back to 4)

Ok, so what are the things that can be a problem here?

a) Sending render data to the slave takes very long or doesn't "get through" to the slave.
b) Sending assignments to the slave takes very long or doesn't "get through" to the slave.
c) The slave start working on the assignment, but its GPUs crash causing it to never return any results to the master.
d) Sending back results to the master takes too long, causing the slave to run out of assigned work long before it gets new work, i.e. it's sitting there most of the time waiting for new work.
e) Something else... maybe the plugin does something special which causes the network rendering to stall.

Your last screenshot indicates a GPU crash on the slave. Does it happen reliably? I.e. if you stop the daemon via CTRL-C and then restart it using the _run_installed_daemon.bat and then using it for network rendering, does it immediately crash?

On the other hand, you are saying that it works in the Live Viewer. There is probably one major difference between rendering in the Live and the Picture Viewer: In the Live Viewer images are usually rendered as LDR results, i.e. they are only 1/4 the size than the HDR results which you are probably rendering in the Picture Viewer. Another difference is that you probably don't have any render passes in the Live Viewer, but do have them in the Picture Viewer. Render passes require more memory on the GPUs and take longer to transmit to the master.

Btw, which OS and which version are you using? What is the resolution you are rendering? Is it an animation or just a single frame? Does the rendering finish, i.e. reach the maximum samples/pixel?

One thing you can do to help me understand the problem better is to enable logging on the slave and the master, which is described here: viewtopic.php?p=234275#p234275

The same way you can enable logging for the Standalone (master), you can do it for the plugin (master) and for the slave. You just have to copy the octane_log_flags.txt file into the directory where the octane.dll is. Make sure that this is in a directory where you have write permissions (otherwise Octane can't write the log files). To kick off logging the application needs to be restarted (because the flags file is only read once in the beginning).

That's it for a start. I hope with the log files I will see what's going on.
Octane For Cinema 4D developer / 3d generalist

3930k / 16gb / 780ti + 1070/1080 / psu 1600w / numerous hw
User avatar
aoktar
Octane Plugin Developer
Octane Plugin Developer
 
Posts: 15962
Joined: Tue Mar 23, 2010 8:28 pm
Location: Türkiye

Return to Help / Tutorials


Who is online

Users browsing this forum: No registered users and 10 guests

Fri Apr 19, 2024 10:54 pm [ UTC ]