Another Mac Pro user is looking for a (direct) GPU-Cluster

[serge-m]

Hi All,

I'm interested in buying (or building) a GPU cluster like the ones from Amfeltec (http://amfeltec.com/products/gpu-oriented-cluster/) or Netstor (http://www.netstor.com.tw/_03/03_02.php?MTEx#).
I've read all (I think) possible posts here in this forum or over at macrumors.com, that covers this subject and found out, that Mac Pros have their own way how they handle external (or more than 4) GPUs.
Up until now it was not clear for me, how much GPUs a classic Mac Pro (5,1) can recognize and if it runs well with - lets say - the Netstor layout.

Here is what I want: one Cuda card (like my PC GTX 980, that I have now) in my Mac Pro and 4 or more cards as a cluster outside.
I know, there is a PC build up option via network, but I want the cards directly connected to the Mac.
So, has anyone good experiences with his/her Mac and a Netstor (or Amfeltec) in combination, that is running fast and well ?
I imagine an Octane workflow, that has a fast Live Viewer and a fast render output

Thanks in advanced for any useful hints

Ah and: is OS Sierra able to handle more graphic cards than 3 or 4 ?

[pixelrocket]

I had an Amfeltec chassis and used it with Mac Pro for about a year or so. It worked well, though there was a slight delay in sending large amounts of data over the 1x connection to the chassis. I ended up selling it and building a PC for rendering. It was much easier to manage and use with the network rendering option in Octane and Cinema 4D.

I also ran into issues trying to fill all 4 slots of the chassis. OS X is very particular about what generation Nvidia cards you can use. You an read more about that http://forums.macrumors.com/threads/el- ... t-22865319

As for the OS, you might be limited to 10.11. I don't know if it's possible to run an external GPU chassis in 10.12.

Another factor to consider is the newer Pascal cards. As of now, there's no support for them in OS X. Take into account that the 1070 cards seem to score the same as a 980ti in Octane but run cooler and use less power and are relatively cheap compared to the 1080.

If you want to future-proof your hardware, it would probably be better to build a PC render box. You can still design on the Mac and render on the PC.

[serge-m]

Thank for the answer and sharing your experiences, pixelrocket.

Yes, I've heard, that the Amfeltec option has a lower respond overall - cheaper, but slower...
That's why I probably want to choose the Nestor version. It also supports PCIe 2.0, which is the maximum bandwidth in my case.
Also it seems, that using the same type of cards generates less problems (as I saw in the link you provided).

So, maybe 5 type of cards (1x internally, 4x externally) should work with the Netstor option ?!

[pixelrocket]

I don't have any experience with the Nestor, and I don't know if it will work successfully or not.

Another thing to keep in mind is the heat generated by the GPU's when rendering. With the Amfeltec, you could always put a fan in front of the cluster to keep the temps reasonable. In the Nestor, the case will hold a lot of heat. You might have to run it with the door off to keep the cards from throttling or overheating.

Again, these were issues I dealt with and ultimately decided to build a PC. You are stuck buying used older cards to keep OS X compatibility if you use an external chassis.

By the time you add the price of the Nestor and the GPUs, you can build a really nice PC for rendering. Yes, it's another OS to manage however you can run on new hardware and are not dependent on Apple and Nvidia and a beta web driver.

[serge-m]

Hot damned... it's a difficult time for us Mac Pro users right now...
This circumstance alone makes me think choosing a different renderer.

Thanks again for the hints and explanation, pixelrocket.

What I've heard is, that the Netstor chassis should provide enough air for ventilation. From what I read.
Yes, because of the uncertainty of nvidias driver and Apples goodwill, updating to new cards could be a longer wait.
But at the moment I'm running a single gtx 980 (PC) and have a score of about 97 and anything higher would be satisfying - like this guy with the cubix expander, who's running 5 cards in total successfully.

I just spent €700,- for an Octane render job and the job before another €1.000 !
I would rather invest the money in my own little Mac render device

That is why I asked again: anyone here who is running up to 4-5 cards successfully directly with a cMP ? Fast enough and solid ?

[swellsparky]

I am currently using the Amfeltec GPU Expander Chasis, in a Mac Pro 5,1 and it is working fine ... except ... I can't get the system to start up with more than 2 GPUs in the chassis. I've read it has something to do with OS X not recognizing more than 2 external cards. Currently I'm running 10.12 (Sierra) - but I had the same issues under 10.11 (and 10.10 for that matter) - it's rather frustrating, but at least I have 2 cards working.

I must admit I don't see a huge issue with lag in the Live Viewer tapping into those external cards. I've got 1 980Ti in the MP and 2 in the chassis and it feels pretty snappy. Considering the Amfeltec was $500, much cheaper than other solutions, it's works well. Just wish I could stick another 2 cards in there.
I have 6 other GPU's (all matching 980Ti's) stuck in other computers on my network and are used only at render time using Octane Net Render. Except for some occasional network bottlenecks which crashes slaves my system has worked out well - and has about a 1200 Octane Bench Score.

Like you - I am resisting like mad having to invest in a whole new PC platform, and learning the Windows-world - but my Mac Pro is so old (2010) that it really is about time.

Anyway - wanted to share my experience - which works - (10 GPUs rendering thousands of frames a week) - isn't pretty - but comparatively inexpensive.

Good luck!

[OuterGamut]

Hi all,

I am another mac user looking for options in this direction. Mostly though it seems we are going to have to be OS agnostic in the future. I am parting out a PC to purchase this year because the mac options seem pretty hinky or wildly expensive.

If anyone breaks any ground on the amfeltec node in a cMP please write back into this forum and update us all. It would be amazing to just add 4 gpu's to my rig.

[Collider]

What information are you looking for that hasn't been discussed (search the forums for more, but cliff notes below)?

You can only use up to 3 maxwell architecture gpus (980Ti) in the 5,1 macs, no getting around it. You can use up to 5 of earlier architecture. The amfeltec solutions are nice for getting your cards out of the box, but a more cost effective solution may be the 3m pcie extensions (one card in box, 2 extensions if using 980 Ti - possibly 2 cards in box? I haven't tried that). You will have to buy an octane license (and possibly plugin license depending on how you're rendering) for each machine and figure out a way to place and cool your external cards, so include that in your cost. The hybrid cards might be a nice way to go.

Smicha is doing a fantastic youtube series on building a water-cooled PC if you're interested in that route. His is 14 cards, but the ideas and parts are the same for a 7 card install.

[pixelrocket]

To add to this conversation, as of now, there is no Mac driver for Pascal series cards and there doesn't seem to be one in development either. You will be stuck with 900 series cards or older if you stay with the Mac.

[OuterGamut]

Thanks Collider for summing it up. I have basically gathered all that info, not so concisely though, via the forum.

To me, it is much simpler to add another computer to my studio. The costs will be more but it's an investment in the future. I will be able to have a newer CPU in-house, a net render addition and I can start re-familiarizing myself with windows. The Xeon's in mac pro's are nice but, you know, long in the tooth. I really don't see apple all of the sudden un-smalling their product line and opening up it all to a tower again. Their trajectory is smaller, less custom and less high end.

Liquid cooling is a fun sport to watch, for now I am excited about hybrid cards. I need to check out the 12 card renderer video series, sounds like a mountain of sorts.