Open air frame, expanding GPU rendering. Help!

[FAZ]

Hi guys,
I could use some advice with upgrading our gpu count. We are considering an open air frame that would hold x4 gtx 1080s. PC platform. Looking to get x4 gtx 1080s into a cost friendly open air frame system. I think as of right now, our Apex 4s motherboards do not support more than 4 gpus. So we will probably need a low-end PC rig to host this.

Our current setup is as follows...
Boxx Apex4 with x4 gtx 980 ti
Boxx Apex4 with x2 gtx 980 ti (can only hold two)

Questions...
Does Octane still only support 12 total gpus, or did that cap go up?
What will we need besides the air frame and host card?
Is this even a good route to take? (our space is pretty clean and relatively dust free)
What kind of ram would the host need to act as only a x4 gpu renderer? (ultimately, we were thinking of putting x2 gpus on the host and x4 on the air frame)

Thanks for any info and advice!
-Alex

[BorisGoreta]

I have two open rigs with 6 GPUs each connected over network to master computer. Each rig has super cheap Asrock H81 Pro BTC with the crappiest Intel CPU on it and 16GB of RAM and SSD and that's it. Really cheap. RAM is regular DDR3 RAM. There is no problem with dust and it being open helps cool it down plus you can purchase GPU models with custom coolers like Asus STRIX which are way cooler than the stock format. You would also need 1 or 2 PSUs for each rig, I highly recommend Seasonic since they are completely silent. Asus STRIX cards are also completely silent when not rendering ( fans are not spinning at all ) which is nice for keeping noise low. Octane supports more than 12 GPUs now. You shold look it up yourself I am not sure, could be 21 or more now. You don't need a host card because host computer communicates with the slave render nodes over Ethernet. I am not sure about GTX1080 since they are at the same speed as GTX980Ti which are much cheaper.

[Notiusweb]

Just my 2 cents

Questions...
Does Octane still only support 12 total gpus, or did that cap go up?

Octane 2 - 12 GPU Max
Octane 3 - 20 GPU Max (per rendering session, whether any combo of networking nodes or single rig expansions)

What will we need besides the air frame and host card?

-PSUs for the video cards
-possibly extension chords, either 14 or 12 Guage, to route a PSU to another outlet on a separate circuit.
(This is if power usage on a single outlet trips breaker, you can consider connecting one PSU via an extension chord to another outlet on a separate circuit switch on the main breaker.)
-Fans, although external fans usually only cut temp at most by 5 degrees.

[seltzdesign]

I wouldn't go for 980ti for maximum GPU count (of course unless you absolutely need the 6GB of memory). They are too expensive, even for used ones. Get the much cheaper 780ti. 980ti has Octane bench result of 125 and 780ti has 103, so 980ti is only 25% faster, but costs around 100% more.

I have rencently bought 3 780ti's for around 170 Euros each and a grand total of around 500 Euros for Octane Result of >300, so still more than 2x980ti, which would have cost me at least 800 Euros. If you are going with many GPUs then the best value for money is always going to beat slightly better performance.

There is a handy little chart here for comparing best value for money: viewtopic.php?f=21&t=56482&p=289797#p289797

[seltzdesign]

I have two open rigs with 6 GPUs each connected over network to master computer. Each rig has super cheap Asrock H81 Pro BTC with the crappiest Intel CPU on it and 16GB of RAM and SSD and that's it.

That sounds good, but how are you connecting 6 GPUs on a motherboard with only 1 PCIe slot? Can you connect them to the PCIe x1 ports? Are you using PCIe extension cables?

[BorisGoreta]

Yes, I am using connectors of this type:

https://www.amazon.de/gp/product/B01H4P ... UTF8&psc=1

Since PCIe x 1 is many times faster then the Ethernet connection used to communicate to the master computer PCIe x 1 interface is not a bottleneck.
Ethernet is cca. 112 MB/s and PCIe x 1 is 500 MB/s ( minus some overhead ).

[seltzdesign]

Yes, I am using connectors of this type:

https://www.amazon.de/gp/product/B01H4P ... UTF8&psc=1

Since PCIe x 1 is many times faster then the Ethernet connection used to communicate to the master computer PCIe x 1 interface is not a bottleneck.
Ethernet is cca. 112 MB/s and PCIe x 1 is 500 MB/s ( minus some overhead ).

Wow, ok, thats really interesting. I was always under the assumption you would need faster connection speed, but you are right about the difference in speed. Maybe its counter-intuitive what the GPU is actually doing when and how much data gets sent between the CPU and GPU.

So Octane just sends the scene data and rendering instructions to the GPU (not so much data), GPU does the sample rendering, sends calculated Sample back to Octane (via CPU)? Is that roughly what is happening?

Am I right in thinking there is therefore no significant dropoff in render time even using PCIe 1x? Have you run Octanebench to verify this?

Interestingly those USB adapters are actually much cheaper than the PCIe 16x extender cables and of course you can get any length and you could even have them externally. Hmm... 4 gpu's in the main box and being able to have another case with another 4 gpu's with own powersupply and being able to just plug in 4 USB cables and one to sync the PSU's.. *goes back to drawing board*

[BorisGoreta]

Here is basically how it goes, please correct me if I am wrong but I think it is important to know what is going on in order to know where you could improve performance and at what cost.

1.You load up your scene and tell Octane to start rendering.
2.Octane starts compiling/preparing the scene for rendering.
- this is all data pushing and CPU work at the workstation end, this part is multithreaded but not 100%, I get peaks at 30% CPU usage on a 16 physical core machine
- you could optimize this part by having higher clocked single CPU with less cores. Anything better that this (like dual Xeon systems ) would not be fully utilized by Octane.
3.Octane starts rendering on the workstation and starts sending rendering data to the slaves via network
- here your network connection with the slaves comes into play
- usual 1Gbps ethernet interfaces pushes data at cca. 110MB/s, data sent to the slave is compressed prior to sending so this helps a little
- having 1x PCIe speed at the slave is more than enough to receive data at this rate
- you could improve this part by having multiple 1 Gbps interfaces sending data to the same slave
- or you could invest in 10GbE network cards which would speed up the transfer to more than 750MB/s which would match or exceed the 1 x PCIe theoretical max speed at 512MB/s
4.After all data is pushed to the slave, the slave starts rendering and occasionally sends back rendering data

It is important to understand that PCIe or Ethernet speed does not affect GPU rendering speed once all data is pushed to the GPU memory. Usually when working with the scene you push majority of the data when you fire Octane and after that Octane updates just the changes you make to the scene which is easily handled even by the slowest network and PCI connections. Or if you render a final sequence all data is pushed to the slave GPUs when rendering the first frame, usually there is no delay at the slave end in consecutive frames since all data is already there.

Octane bench does not work with slaves, only local GPUs.

Cases when you could notice slowdowns when using network slaves for rendering:
huge scenes, close to 6 GB or more, it takes a while to transfer this data to the slave using 1Gbps ethernet.
scenes that have large changes from frame to frame ( for instance large 10 million poly ocean animation where it has to change geometry every frame )

But otherwise you wouldn't notice at all that you have network slave running instead of local GPUs. It really works seamlessly and very, very well.







Basically when the scene is compiled and prepared and starts rendering on the master computer, Octane sends all this prepared data to the node.



In this moment you need fast network connection to the node. Default connection is 1Gbps ethernet because all motherboards have them nowdays.
You can improve this by buying 10GB network cards on both ends plus possibly a 10GB switch if you need to connect more nodes this way.

[FAZ]

So much great info here guys! Thank you.

[seltzdesign]

.. It is important to understand that PCIe or Ethernet speed does not affect GPU rendering speed once all data is pushed to the GPU memory.

Thanks Boris, that was the important part. Good to know, but now that I think about it, it kind of makes total sense. I have been using network rendering as well, but still assumed you would need faster PCIe connection for the GPU to work at full speed.

Have you run Octanebench on the slave though? I am actually using network rendering just for working and for rendering final animations I start them on the slave directly, because the computer I work on I have to switch between Mac and Windows, so cant leave it running in the background. I am of the opionion that for our really small setup, its better to have one render node with as many GPUs as possible compared to 2 lower powered ones with less GPUs. So knowing you can use PCIe 1x to connect a GPU is kind of a revelation.

If only there were some nicer cases for things like that. It seems like the only 2 possible routes are a total open case like bitcoin miners use or a huge pc case. Might have to build my own.. because there is nothing that I am aware of that is a) enclosed, b) has room for 6-7 double-width GPUs, but not on the motherboard c) has smart ventilation, ie. cold air comes in at front or bottom and goes out back or top, d) has room for 2-3 PSUs. There was that one by Red Harbinger (https://www.indiegogo.com/projects/dopa ... arbinger#/), but its not available anywhere.