External PCIe Expansion options

[Raoul]

Hi All,

I was hoping some people could help me out on recommendations for an external PCIe chassis.
Apologies if this has been asked elsewhere and i missed it.

I currently have a 2012 Mac Pro tower with a 780ti and a 680ti running internally. osx 10.9.5

I wanted to expand the rendering capability and have been looking at external gpu solutions.
These options seem to be the most often noted.

http://www.cubix.com/catalog/buy-xpander-desktop

and

http://www.netstor.com.tw/_03/03_02.php?MTEx

I'd like to know the following:

A) any problems/pittfalls people are aware of with these systems?

B) any additional / better / cheaper ways to achieve the same thing?

C) will they support the next gen cards to come through like Nividia gtx 1080ti?

D) if you load it with 4 titans for example will heat/noise be an issue

E) any Australian distributors that would sell these?

F) would it just be cheaper to buy a pc or another route?


If anyone could answer one or more of these Q's that'd be awesome!
Thanks!
R

[glimpse]

Hi Raoul,

there are some answers scattered around the forums, but I'll try to distil thoso & put them into one reply.

Netstor & Cubix seems to be some of the best known units. Internaly those are very simillar as they are prety simple. You get the signal from Host board & then PLX chip "splits" it allowing You to use more PCIe devices.

One thing to know si that most likelly You'll be limited by Your operating system rather than hardware, 5-6 GPUs is the most I've seen being done & it's pretty tricky.. as they will not work with just any version of OS.

So to reply Your questions (feel free to ask more details):

A. it's easy to setup & should be plug & play more or less.. jsut need to be carefull with OS &.. You better choose cards acordingly.. 4+1 or so..- do some googling, otherwise You might have problems to boot or recognise those cards.

B. You get waht You pay, but it's possible to get backplate, PSU, case (bellow 1K$) & plug pair of GPUs into it (latelly built a piece based on that)..If You wish to have 4GPUs, You need Single Slot cards with this option.

C. again, the support comes from OS first of all..as for cards we do not have them..&it depends what type of conector they are goign to have & if it's going to be backwards compatible (most likely yes, but..let's wait few days to see =)

D. I'm not too much into cubix, but that Netstor box have slightly wider gaps between PCI slots, meaning cards have more room to breath =) this allows to spin fan at lower speed, what leads to lower noise output. Over all, it's not the most silent solution & temps should stay ~70-80C with 4x TitanX, but it's far from worst also =)

E. Netstor comes from Taiwan, so jsut google around to see how to get the cheapes thing - I'm pretty sure You should be able to find something in AUD for ~ or less than 3K. Yes it's not cheap, what brings us to last point..price & value.

F. to build a slave would be devinitelly not more expensive.. as for 2-3kAUD You can get 6Core 40lanes CPU + mother board that supports 4 or even up to 7GPUs, Good PSU + some RAM. This route would cost You the same, but You can use this machine as slave or backup unit.

Overall eGPU is not the only option..& Mac are not so friendly with Multiple GPUs. Getting Slave or simply changing Your workstation might be the best deal & I would go to eGPU if ..You do not mind to have some headache &..really do not want to switch to windows..

Hope that helps =)

tom

[Raoul]

Awesome, Thanks Tom!,
I've seen your site a number of times, thank you for the research and effort! Its a pretty murky path to walk for us less technically proficient out there.

Okay.. so just wanted to clarify a few things:

You suggest a netstor over a cubix.. in that theres more breathing space between the cards? any other reasons?

I've read a lot about OSX gpu limitations but theres lots of contradictions. Some say El capitan will run 4 external if they are the same card other say it won't..
But i take i do need to upgrade to el capitan as a basic requirement. ( was holding off for other software conflicts with this OS).

Do you have any experience with an exact setup for a classic mac pro that has proven to run 4 cards external.. + the one internal card.
If so, would love to know the exact details so i can replicate?

I guess i'd like to stick to something i can plugin into the mac.. so i don't need to run another windows system,.. theres added software costs with that route. But will look down that route if i can find a reliable way to run 4 external gpus with a mac pro.

Thanks so much for the help! i wish there was a central and regularly updated database of what does and doesn't work.

Cheers!

R

[smicha]

Tom,

Your knowledge on external (and not only) solutions is impressive.

[glimpse]

Hi, Raoul,

let's clear/polish some points.

"You suggest a netstor over a cubix.. in that theres more breathing space between the cards? any other reasons?"

both Netstor & Cubix have extra space as far as I know..Main reason why I've talked about Netstor..is because You getting the same for less..so I would rise the question why to get Cubix? (I know they have separate products. some software, etc. but it's not usefull for us) so, why to pay more for Cubix, Magma, One Stop, or any other solution..

"I've read a lot about OSX gpu limitations but theres lots of contradictions. Some say El capitan will run 4 external if they are the same card other say it won't.. "

as mentioned before, that mainly depends on OS version, drivers, cuda..& then a bit on thin air (sometimes it's very hard to know what is messed up, but it's MAC related..) I'll attached some pics bellow if You are interested in exact version we've made to work for eGPU with 4GPU box I've showed.

"But i take i do need to upgrade to el capitan as a basic requirement. ( was holding off for other software conflicts with this OS)."

if You need that flexibility for the software to work.. Just build a slave & do not worry anymore about Mac issues with hardware..-it's another layer of complexity & I doubt You need it.

"Do you have any experience with an exact setup for a classic mac pro that has proven to run 4 cards external.. + the one internal card. If so, would love to know the exact details so i can replicate? "

I've added printscreen & few shots of the rig. You can try to replicate something simillar or..use more expensive, but faster (You get what You pay) enclosure from let's say Netstor (or any other company if You willing to overpay =)

"I guess i'd like to stick to something i can plugin into the mac.. so i don't need to run another windows system,.. theres added software costs with that route. But will look down that route if i can find a reliable way to run 4 external gpus with a mac pro. "

it should be as ease as plug & play & it is on windows, but not on Macs, sorry to say.. You will msot likelly have some issues solve unless You're extremelly lucky.

So any options? As mentioned in previous reply, I would try to build eGPU but then switch to render node if that eGPU would not work. I guess You could sell backplate prety easy.

Render node costs in terms of software is not that much.. OEM licence of windows You can get for 20-30$ I guess & then..OctaneRender Standalone csots what 150-200$? So what are the other cost? quad core CPU will be around 200-300$, good AIO cooler ~100-150$, then 32GB of RAM, let's say DDR3 is around 150$ & motherboard.. with 4x 8lanes.. probably 200-300$ ..ou PSU let's say 300$ & case 100-200$ - I mean in total You're looking 1000-1500$ for complete system (including licences) & then simply pick GPUs of Your choise.

Then again, I'm not to say slaves does not have any downsides..comunication between slave & Your Mac, compared to eGPU solution is not nearly as fast..but if eGPU cause You might have more headache with compatability..

So in the end, none of these solutions are perfect, Raoul. I mean having eGPU has ups/downs, then slave as well..& it really depends what are Your preferences are.

" I wish there was a central and regularly updated database of what does and doesn't work. "

it's hard to maintain this type database..There's a lot of good information scatered in these forums for instance, but it's just crumbs..

I have this idea to update site & have specific section for GPUs (ins & outs about different cards on the market), other for eGPU solutions (from the shelf, DIY, Mac related, etc), then Complete systems (different routes You can take), Watercooling..-we'll see.

Tom, Your knowledge on external (and not only) solutions is impressive.

Thanks Sebastian.

by the way we chose 4x GPUs in external enclosure 'cos we got good price for them (bought new 780 6GB for 320EU & it gives like ~80 in OctaneBench out of the box, up to 100 if OCed) from value perspective that was a good choise. We kept OC out of the way to make sure that stability is not influenced by that. Alternativelly You can stick with pair of 980Ti, to see if You can make them work on Your setup - those would give ~ 240 , maybe 250 together & prices are dropping (if nvidia is goign to realease anything new they will dive even more). Avoid new cards for now.. as it will take time for glitches to be cleaned out in drivers & Mac OS compatability to arrive..Amfeltecs backplate seem to be value oriented option as well, but keep in mind that 1x limitation.

[pixelrocket]

I thought I might add to this since I am currently running a Mac Pro 5,1 and using Octane with Cinema 4D. My current GFX card setup consists of a GTX 680, and two GTX 980 ti cards using a Amfeltec GPU Cluster http://amfeltec.com/products/gpu-oriented-cluster/. While the cluster works great, the transfer time on larger scenes is noticeable and the feedback from the display is not nearly as quick as having all of the cards on one 16x or 8x connection. I normally leave My 680 card off for rendering so my display is still fast, plus the RAM isn't the same size as the 980 cards.

Here's a few things to think about when using Octane on the Mac...

The good:
You still get to design on a Mac, and use OSX.
Octane works well enough get most levels of production work done.
You can use some of the newer cards and up to 5 GFX cards if you are running OSX 10.11

The Bad:
Your connection speed to the cards will be slow using an external chassis.
If you want to run two fast cards internally (2 x 980ti) you will need an external power supply.
I was never able to get more than 4 cards TOTAL working with the chassis (1 internal and 3 on the chassis), and you have to be careful if you mix chipsets (780 series w/ 980 series) as you can't run 5 x 980ti in OSX currently.
Here's a thread about GPU issues / limitations in OSX 10.11 http://forums.macrumors.com/threads/el- ... t-22130114

Once I really got into using Octane on a daily basis, I quickly realized that I needed more horsepower and built a dedicated PC for rendering. I'm running a Windows machine with 4 x 980ti's water cooled as a render slave / occasional workstation.

In my current workflow I still use the mac for designing, and when I'm ready to render I send the file to the PC via Team Render. This isn't the most efficient use of the PC, but I have a LOT of mac software, and I still like designing on the Mac. This way, I can keep working and the files get rendered really quickly in the background. There are times I network all the cards together for one render but most of the time I just let the PC run in the background.

My advice, build a PC. Tom is right, it will be cheaper and you can get a lot of horsepower for the money if you build it right. It will scale much better than the Mac, and you can always swap out components when newer cards come out. Also, take a look at water cooling if you decide to go with a PC. Running the cards on water vs air is a night and day difference, and Tom or Smicha would be the ones to talk to if you decide to go that route. Their posts on building and cooling PC's were a great resource when I built mine. This was my first water cooling build and it really isn't that bad. There is a TON of info on this board and on places like overclock.net for water cooling. Even running two hybrid 980ti's would be better than running two of them on air.

[BorisGoreta]

I have Netstor box for around 3 years now I think and it is a rock solid product. Spacing is larger between GPUs then in standard motherboards so you get more ventilation.

When I purchased it I made 2 modifications. I pulled out the chipset fan because it produced a bit of noise which I didn't fancy. The PSU that goes with it is very noisy so I swapped it with Seasonic. Other that that as I said it is fantastic.

[Raoul]

Ahh Awesome help everyone!

Boris: Great to hear the Netstor is working out for you! Thats what i've gone for now.. Just waiting for it to ship.. And am now deciding on what cards to put in it.
Can i ask what GPUS you have in yours? and what you have in the mac or pc its connected to?
I currently have a flashed 680ti and two standard 780tis one of which is just gathering dust..

Pixelrocket: which brings me to you... (great info btw thanks!)
You say that you can run 5 cards with 10.11 .. is that 5 cuda cards? or one standard card and 4 cuda...
But you also say they cant all be maxwell generation. (i'd read that same article you linked to but wasn't 100% on details)
So would you suggest i use my 680ti for the monitor.. then 2x780tis and purchase an additional 2x980tis ?

The PC option does sound pretty appealing.. But I'm not currently in a position to do the build myself.. and finding someone over here in Melbourne.. to build me a water cooled 4 cards pc render box is going to be difficult. Theres also the added cost of software cinema/octane on that box ( is that right? - extra license?)..
But my thinking is... if i want to expand beyond 4 cards... i buy the PC down the line.. and swap the netstor between computers as necessary.

On top of all that.. theres the upcoming pascal 1080's. deadline for upcoming job hits just before the release date so i guess i'll have to just wait and upgrade futher down the line.


Do you know if the Amalftec is the same basic components and connection as the netstor.. should i expect a similar slow down in sending scenes to render?

Cheers all.. been bery helpful.
R

[BorisGoreta]

All are regular factory air cooled TITANs. Vanilla, Black and X mixed. I would recommend such models because air should flow from back to front, Netstor is still too small to accomodate cards with custom coolers ( dual or triple fan on top).

[pixelrocket]

Ahh Awesome help everyone!

Boris: Great to hear the Netstor is working out for you! Thats what i've gone for now.. Just waiting for it to ship.. And am now deciding on what cards to put in it.
Can i ask what GPUS you have in yours? and what you have in the mac or pc its connected to?
I currently have a flashed 680ti and two standard 780tis one of which is just gathering dust..

Pixelrocket: which brings me to you... (great info btw thanks!)
You say that you can run 5 cards with 10.11 .. is that 5 cuda cards? or one standard card and 4 cuda... R

Supposedly you can run 5 cards (all Cuda) however the MacRumors article was the only resource I could find with someone actually getting it to work. I was running 4 cuda cards at one time, and never could get 5 to work.

But you also say they cant all be maxwell generation. (i'd read that same article you linked to but wasn't 100% on details)
So would you suggest i use my 680ti for the monitor.. then 2x780tis and purchase an additional 2x980tis ?

Keep in mind, if you run 780tis and 980ti's you won't be able to use all the ram on the 980's because Octane will use the lowest amount of ram between the cards so you will be wasting the extra ram on the 980ti's. If you want to run the 780ti's, you would be better served either buying two more 780's or selling the two you have and buying the 980ti's. Take a look at the octane bench results and see which gives you the best bang for the buck. The octane bench scores between the two cards are really close, so it may be cheaper to just add more 780's.

The PC option does sound pretty appealing.. But I'm not currently in a position to do the build myself.. and finding someone over here in Melbourne.. to build me a water cooled 4 cards pc render box is going to be difficult. Theres also the added cost of software cinema/octane on that box ( is that right? - extra license?)..
But my thinking is... if i want to expand beyond 4 cards... i buy the PC down the line.. and swap the netstor between computers as necessary.

I'm not gonna lie, building my first PC was a HUGE learning experience compared to working with a Mac on a daily basis. But, if you have any technical knowledge at all, and can follow instructions, you can build a PC. I built my first one just a few years ago. The great thing about the PC is that you can add parts as you go. Start out with a good case and Motherboard, and then add cards along the way. You can run hybrid cards if you don't want to build a full custom water cooling loop. It really is easy, and there is plenty of info on the web for building your first PC.

Yes, you do have to have a license of Octane for the render slave, and if you want to run Net Render, you will need an extra plugin for C4D.

The Nestor will have a faster connection to the chassis than the Amfeltec. I think the Nestor uses SAS, which will be much faster than the 1x PCIE that the Amfeltec uses. But, you have to run reference GPU's in the Nestor vs an upgraded / overclocked card with a better cooler in the Amfeltec. If you run the Amfeltec, you can run any card you want. Here is a blog post from a shop running an Amfeltec along with the GPU's in their PC workstation. http://elasticpictures.com/1662/octane- ... expansion/

On top of all that.. theres the upcoming pascal 1080's. deadline for upcoming job hits just before the release date so i guess i'll have to just wait and upgrade futher down the line.

I was in the same situation, waiting on the new GPU's. I decided to bite the bullet and I'm glad I did. Work needs to be done now, and faster cards in the future weren't helping me get stuff done today. The extra horsepower has really come in handy on large jobs. I can always sell the cards when I get ready to upgrade. Not to mention from the looks of it, we may not get a shipping 1080 card till Q3 and who knows how much faster they will really be.

Do you know if the Amelftec is the same basic components and connection as the netstor.. should i expect a similar slow down in sending scenes to render?

See my previous comments about the speed difference. The Amfeltec is cheap and it works well for what it does. It's not a replacement for the Nestor, but at least in the U.S., it's about a quarter to a third of the price of the Nestor. The Nestor has a much faster connection but is not cheap. You can build a nice PC for the price of the Nestor Chassis as well.

Cheers all.. been bery helpful.
R