What if I use more than 20 GPUs

[mbutler2]

Follow up note: I had trouble getting two masters with one slave each working concurrently. So for overnight renders, it's just one master controlling two slaves. The other machine just rendering on its own.

[mbutler2]

I'm also noticing that it's not worth it to run so many GPUs at the same time. Still testing, but it seems that lots of time is lost coordinating so many GPUs over the network. At some point you're better off running multiple separate renders in parallel, rather than a master with too many slaves.

[Goldorak]

I'm also noticing that it's not worth it to run so many GPUs at the same time. Still testing, but it seems that lots of time is lost coordinating so many GPUs over the network. At some point you're better off running multiple separate renders in parallel, rather than a master with too many slaves.

Yes, it is also going to save you time for mult-frame/segment rendering, as we do on ORC.

[Seekerfinder]

Just a data point: we are in a pac bell building with nearly unlimited power and decent cooling. We still strugglw to get more than 20x 1080Tis working reliably...

Hi Goldorak,
I am curious as to what the limitation is with regards to power & cooling you were referring to here(?) I understand these requirements with respect to multiple GPUs but surely we're talking a master & local slave setup in this case. So both power & heat loads are distributed. Could you perhaps clarify?

Thanks,
Seeker

[Goldorak]

Just a data point: we are in a pac bell building with nearly unlimited power and decent cooling. We still strugglw to get more than 20x 1080Tis working reliably...

Hi Goldorak,
I am curious as to what the limitation is with regards to power & cooling you were referring to here(?) I understand these requirements with respect to multiple GPUs but surely we're talking a master & local slave setup in this case. So both power & heat loads are distributed. Could you perhaps clarify?

Thanks,
Seeker

We have DC mini-room we use for internal rendering in one of our office units using 8x Linux GPU boxes - and it's given us a lot of datapoints to consider for real world use outside of a full DC.

In any office setup, while it may be in theory possible to go higher and higher with more 4U boxes x more racks - at some point power our entire floor isn't enough to take us to simple DC levels, and even before we hit that threshold we see reliability starts to drop really fast if A/C can't keep up (i.e. heat wave hits LA, even telecom building power and cooling can be affected).

That doesn't mean it isn't possible to create DC like power and cooling if you have a unique setup for non ad-hoc render farm, which is why we have Enterprise licensing team that can work with customers that really need to go past 20 GPUs per rendered frame.

[Seekerfinder]

Just a data point: we are in a pac bell building with nearly unlimited power and decent cooling. We still strugglw to get more than 20x 1080Tis working reliably...

Hi Goldorak,
I am curious as to what the limitation is with regards to power & cooling you were referring to here(?) I understand these requirements with respect to multiple GPUs but surely we're talking a master & local slave setup in this case. So both power & heat loads are distributed. Could you perhaps clarify?

Thanks,
Seeker

We have DC mini-room we use for internal rendering in one of our office units using 8x Linux GPU boxes - and it's given us a lot of datapoints to consider for real world use outside of a full DC.

In any office setup, while it may be in theory possible to go higher and higher with more 4U boxes x more racks - at some point power our entire floor isn't enough to take us to simple DC levels, and even before we hit that threshold we see reliability starts to drop really fast if A/C can't keep up (i.e. heat wave hits LA, even telecom building power and cooling can be affected).

That doesn't mean it isn't possible to create DC like power and cooling if you have a unique setup for non ad-hoc render farm, which is why we have Enterprise licensing team that can work with customers that really need to go past 20 GPUs per rendered frame.

Makes sense, thanks Goldorak.
Best,
Seeker

[Lewis]

HI Goldorak, very nice info, thank you.

I have Question.

Is it possible for you guys to enable (feature request) Octane to enable multiframe rendering ? i.e. when rendering machine master send data to all available GPUs on the start and over network then give it option to render frame per GPU instead all GPUs single frame. Basically same as ORC is doing but locally on master+slave machines ? I have feeling that could make network data transfers less intensive since not all GPUs are always same type/speed so after each one renders first frame it will probably be in different time so then each new frame would send less data to free GPU(s) and by that have it done faster since CPUs is anyways sitting idle most of time while rendering. This way it could get CPU busy more during rendering.

Also with that system if any of GPUS fail we would loose only that frame and not stop rendering like currently when network slave fails and waits for user interaction to continue. Then if any frame missing/failed when new set of frame set in CPU Que list then just check for missing frames and reassign missing frame to any available/idle GPU in chain. That's how most Network Render controllers work nowdays.

cheers