Best Practices For Building A Multiple GPU System

[smicha]

IF

SKU-078-44 GPU oriented PCIe Cluster (5ft cable) 1-2 $ 1030.69 USD
[4 Clusters (16 GPU) + 4 channel Host board]

gives you ability to connect 16 gpus the price is indeed attractive.

Just wondering what they say about the 7-8 gpu limit on certain types of mobos - do they mean that even with their expansion one cannot exceed the limit? Let's assume I have a mobo with 7 pci express slots and to every slot there is SKU-078-44 connected - giving 28 overall slots for graphics cards available (56 gpus on 28 titans Z). Will any system (Linux/Win7/XP) see all these cards? What do you think guys?

[Seekerfinder]

Just wondering what they say about the 7-8 gpu limit on certain types of mobos - do they mean that even with their expansion one cannot exceed the limit? Let's assume I have a mobo with 7 pci express slots and to every slot there is SKU-078-44 connected - giving 28 overall slots for graphics cards available (56 gpus on 28 titans Z). Will any system (Linux/Win7/XP) see all these cards? What do you think guys?

Highly unlikely that it would see that many. There are probably RAM allocations associated with available lanes and the OS / BIOS will have 'some' limit and who knows what other possible complications. But, that's just speculation because I do not have the hardware to test the limits. And what frustrates me is that we're all still speculating on the GPU limit.

Otoy is not being helpful in this regard by confirming the GPU limit of Octane on a single machine despite many user requests for this information and frankly, as a GPU software developer, I'm not sure that it's fair of them to withhold such a vital piece of info. And no one seems to be testing the limits. 9 GPU's is the current reported limit (on XP). But that's probably because the user has only that many GPU's available...

Anyone with splitters, an extra PSU or 2 and 10 GPU's willing to test this for us?

Seeker

[Tutor]

Just wondering what they say about the 7-8 gpu limit on certain types of mobos - do they mean that even with their expansion one cannot exceed the limit? Let's assume I have a mobo with 7 pci express slots and to every slot there is SKU-078-44 connected - giving 28 overall slots for graphics cards available (56 gpus on 28 titans Z). Will any system (Linux/Win7/XP) see all these cards? What do you think guys?

Highly unlikely that it would see that many. There are probably RAM allocations associated with available lanes and the OS / BIOS will have 'some' limit and who knows what other possible complications. But, that's just speculation because I do not have the hardware to test the limits. And what frustrates me is that we're all still speculating on the GPU limit.

Otoy is not being helpful in this regard by confirming the GPU limit of Octane on a single machine despite many user requests for this information and frankly, as a GPU software developer, I'm not sure that it's fair of them to withhold such a vital piece of info. And no one seems to be testing the limits. 9 GPU's is the current reported limit (on XP). But that's probably because the user has only that many GPU's available...

Anyone with splitters, an extra PSU or 2 and 10 GPU's willing to test this for us?

Seeker

Hello Seeker, I hope that you're having a good New Year.

I'm the guinea pig on the question of up to 14 GPUs in one system, which is what my Tyan Server is holding now - awaiting my completion of the water-cooling system to boot them all up. My system has 6 Titan Z's (12 GPUs) + a Titan Black (one GPU) + a GTX 780 6G (one GPU) (all now hydros and fourteen GPUs in total). So I should have answers to some of these type questions regarding the number of GPUs - within the next few days. But that question raised by Amfeltec's opening response to my initial request for a price quote, now has me wondering (1) how can one truly determine whether Amfeltec's chassis solutions will work with X no. of GPUs over 7-8 on which particular motherboards and (2) whether Amfeltec's pointing out to me that most motherboards may have a 7-8 GPU limit, is in fact true and whether that is the source of statements/rumors we've been discussing more and more frequently about whether it's Octane or the OS being limited to 7-8 GPUs. It may, in fact, be only (and/or decisions made by SW providers like Otoy grounded in the fact that) the current hardware which most 3d rendering users choose to run there software on can support only 7-8 GPUs. Frankly, before Amfeltec's response, I hadn't given any significant thought about hardware limitations being the basis/or justification for such limits. Curiously, why is Amfeltec marketing a 16 GPU solution and to users of what motherboards, running what OS and and application(s) that take advantage of 16 GPUs? Accordingly, this is what I'm asking the Amfeltec's sales team member who gave me the price quote:

"I forgot to ask you a question that I have regarding your cautionary statement about general purpose motherboards being able to support a maximum of 7-8 GPUs. Are you aware on any particular motherboard(s) capable of supporting up to 16 GPUs and, if so, who is/are the manufacturer(s) and what is/are the motherboard model(s) that you are aware of that can handle that many GPUs? Since I’m a 3d animator who uses GPUs for 3d rendering, that information is very important to my purchasing decisions."

I'll let all of you know what response I receive.

[Seekerfinder]

Just wondering what they say about the 7-8 gpu limit on certain types of mobos - do they mean that even with their expansion one cannot exceed the limit? Let's assume I have a mobo with 7 pci express slots and to every slot there is SKU-078-44 connected - giving 28 overall slots for graphics cards available (56 gpus on 28 titans Z). Will any system (Linux/Win7/XP) see all these cards? What do you think guys?

Highly unlikely that it would see that many. There are probably RAM allocations associated with available lanes and the OS / BIOS will have 'some' limit and who knows what other possible complications. But, that's just speculation because I do not have the hardware to test the limits. And what frustrates me is that we're all still speculating on the GPU limit.

Otoy is not being helpful in this regard by confirming the GPU limit of Octane on a single machine despite many user requests for this information and frankly, as a GPU software developer, I'm not sure that it's fair of them to withhold such a vital piece of info. And no one seems to be testing the limits. 9 GPU's is the current reported limit (on XP). But that's probably because the user has only that many GPU's available...

Anyone with splitters, an extra PSU or 2 and 10 GPU's willing to test this for us?

Seeker

Hello Seeker, I hope that you're having a good New Year.

I'm the guinea pig on the question of up to 14 GPUs in one system, which is what my Tyan Server is holding now - awaiting my completion of the water-cooling system to boot them all up. My system has 6 Titan Z's (12 GPUs) + a Titan Black (one GPU) + a GTX 780 6G (one GPU) (all now hydros and fourteen GPUs in total). So I should have answers to some of these type questions regarding the number of GPUs - within the next few days. But that question raised by Amfeltec's opening response to my initial request for a price quote, now has me wondering (1) how can one truly determine whether Amfeltec's chassis solutions will work with X no. of GPUs over 7-8 on which particular motherboards and (2) whether Amfeltec's pointing out to me that most motherboards may have a 7-8 GPU limit, is in fact true and whether that is the source of statements/rumors we've been discussing more and more frequently about whether it's Octane or the OS being limited to 7-8 GPUs. It may, in fact, be only (and/or decisions made by SW providers like Otoy grounded in the fact that) the current hardware which most 3d rendering users choose to run there software on can support only 7-8 GPUs. Frankly, before Amfeltec's response, I hadn't given any significant thought about hardware limitations being the basis/or justification for such limits. Curiously, why is Amfeltec marketing a 16 GPU solution and to users of what motherboards, running what OS and and application(s) that take advantage of 16 GPUs?

Hi Tutor,
The mobo limit may refer to power the mobo can offer through the pcie slots. But there are ways around that. Amfeltec may not know the limit either. At least one Octane user has proved the 7/8 limit does not exist - at least on his specific old ASUS board - so technically the 7/8 limit statement from Amfeltec is pretty useless for us.

I'm really keen to hear about your findings and thanks for doing the experiment!

2015 is going to be an exceptional year!

Best,
Seeker

[Tutor]

Hi Tutor,
The mobo limit may refer to power the mobo can offer through the pcie slots. But there are ways around that. Amfeltec may not know the limit either. At least one Octane user has proved the 7/8 limit does not exist - at least on his specific old ASUS board - so technically the 7/8 limit statement from Amfeltec is pretty useless for us.

I'm really keen to hear about your findings and thanks for doing the experiment!

2015 is going to be an exceptional year!

Best,
Seeker

Seeker, while you were responding to my post, I edited it to add this:

"Accordingly, this is what I'm asking the Amfeltec's sales team member who gave me the price quote:

'I forgot to ask you a question that I have regarding your cautionary statement about general purpose motherboards being able to support a maximum of 7-8 GPUs. Are you aware on any particular motherboard(s) capable of supporting up to 16 GPUs and, if so, who is/are the manufacturer(s) and what is/are the motherboard model(s) that you are aware of that can handle that many GPUs? Since I’m a 3d animator who uses GPUs for 3d rendering, that information is very important to my purchasing decisions.'

I'll let all of you know what response I receive."

[glimpse]

Amfeltec may not know the limit either. At least one Octane user has proved the 7/8 limit does not exist - at least on his specific old ASUS board - so technically the 7/8 limit statement from Amfeltec is pretty useless for us.

the limit does exist..- & it's not a "useless" as there are plenty of users who have had problems with it.. (& there is very little information where that limit comes from)

http://youtu.be/GOpBlYx2H1o?t=5m - at 5min mark the guy explains that in order to run more than 8 GPUs ASUS have to make custom bios..- this was back then.. hope new EUFI changes something? I'm not sure there..(maybe anyOne tried that)

..but claiming it "limit statement ..is pretty useless" is a bit short seeing - just because You haven't face that problem Yourself, doesn't mean there's no problem. & to further escalate this, I haven't seen much users running more than 8GPUs in this forum..- but threre a plenty of those who have more GPUs..

For OTOY this issue might be ok (not to bother explain), as it's better to have users buying multiple licences than help them building monstrous rigs for rendering.

[Seekerfinder]

Amfeltec may not know the limit either. At least one Octane user has proved the 7/8 limit does not exist - at least on his specific old ASUS board - so technically the 7/8 limit statement from Amfeltec is pretty useless for us.

the limit does exist..- & it's not a "useless" as there are plenty of users who have had problems with it.. (& there is very little information where that limit comes from)

http://youtu.be/GOpBlYx2H1o?t=5m - at 5min mark the guy explains that in order to run more than 8 GPUs ASUS have to make custom bios..- this was back then.. hope new EUFI changes something? I'm not sure there..(maybe anyOne tried that)

..but claiming it "limit statement ..is pretty useless" is a bit short seeing - just because You haven't face that problem Yourself, doesn't mean there's no problem. & to further escalate this, I haven't seen much users running more than 8GPUs in this forum..- but threre a plenty of those who have more GPUs..

Slow down there Glimpse. I did not once say that the limit does not exist, nor that it's not a problem... What I said was that what Amfalec reported to Tutor regarding a 7/8 limit is useless for Octane users, since the 8 GPU has already been broken by at least one user. The currently known limit is 9 GPU's. Make sense? Of course there is a limit and if you read some of my other responses you will see I, and a number of others, have been trying to find this limit - hardware / software / OS / BIOS - and whatever combination. We need credible, useful information at this point.

For OTOY this issue might be ok (not to bother explain), as it's better to have users buying multiple licences than help them building monstrous rigs for rendering.

Yes, that is what seems to be going on. But it does not make it right.

Thanks for sharing the video. I have seen that before. And it sounds as though a key may very well lie in the BIOS.

Seeker

[Tutor]

The prices seem extremely reasonable to me, especially when you consider that the eight GPU cluster costs less than 1/5 of the retail price of one 8 GPU Tyan Barebones Server.

Exactly! The price is very reasonable =) for some users (who preffer to go o air it should be very tempting =)
yesterday watched 'bout ASUS Poseidon GPU with custom cooler. It performs close to strix & on open air rig should not only run higher clock but also lower temps emitting less noise =) now the good point is that You can hook a loop without adding waterblocks =)

Smichas route, going watercooling is good, Your route is good, & this one too - just every case has it's own user & if it would be my money (considering I already have a system, & my needs - rendering stills) I'd go this expander route with 10feet cable to place th cluster in other room =) the tempting point is ability to minimise Your rig =) as even mITX would work just fine =) & there are crazy cases for those..even passivelly cooled =)

Glimpse,

Being in the process of building a water-cooling system that's mainly external, I don't see why one would assume that going the Amfeltec route precludes water cooling.

[Tutor]

Amfeltec may not know the limit either. At least one Octane user has proved the 7/8 limit does not exist - at least on his specific old ASUS board - so technically the 7/8 limit statement from Amfeltec is pretty useless for us.

the limit does exist..- & it's not a "useless" as there are plenty of users who have had problems with it.. (& there is very little information where that limit comes from)

http://youtu.be/GOpBlYx2H1o?t=5m - at 5min mark the guy explains that in order to run more than 8 GPUs ASUS have to make custom bios..- this was back then.. hope new EUFI changes something? I'm not sure there..(maybe anyOne tried that)

..but claiming it "limit statement ..is pretty useless" is a bit short seeing - just because You haven't face that problem Yourself, doesn't mean there's no problem. & to further escalate this, I haven't seen much users running more than 8GPUs in this forum..- but threre a plenty of those who have more GPUs..

Slow down there Glimpse. I did not once say that the limit does not exist, nor that it's not a problem... What I said was that what Amfalec reported to Tutor regarding a 7/8 limit is useless for Octane users, since the 8 GPU has already been broken by at least one user. The currently known limit is 9 GPU's. Make sense? Of course there is a limit and if you read some of my other responses you will see I, and a number of others, have been trying to find this limit - hardware / software / OS / BIOS - and whatever combination. We need credible, useful information at this point.

For OTOY this issue might be ok (not to bother explain), as it's better to have users buying multiple licences than help them building monstrous rigs for rendering.

Yes, that is what seems to be going on. But it does not make it right.

Thanks for sharing the video. I have seen that before. And it sounds as though a key may very well lie in the BIOS.

Seeker

Seeker,
I may be the guinea pig here also because I'm seriously considering purchasing a number of Amfeltec units to consolidate as many of my GPUs as is possible into fewer, but more GPU heavy units. I Intend to maintain my 24 systems to maintain my CPU rendering performance, but I hope to reduce the number of GPU rendering systems to 16 or less. That should save me lots of costs in the long run, while providing me, additionally, with the ability to reinstall on my main motherboards virtually all of my many other PCIe cards that perform other functions. I also haven't given up on my plan to water-cool all of my GPUs, and, in fact, I can foresee going the Amfeltec route even easing that pain. So, I'm mightily pumped up by Rappet and Glimpse's lead.

[Tutor]

Here're the words from Amfeltec:

"The motherboard limitation is for all general purpose motherboards. Some vendors like ASUS supports maximum 7 GPUs, some can support 8.

All GPUs requesting IO space in the limited low 640K RAM. The motherboard BIOS allocated IO space first for the on motherboard peripheral and then the space that left can be allocated for GPUs.

To be able support 7-8 GPUs on the general purpose motherboard sometimes requested disable extra peripherals to free up more IO space for GPUs.

The server type motherboards like Super Micro (for example X9DRX+-F) can support 12-13 GPUs in dual CPU configuration. It is possible because Super Micro use on motherboard peripheral that doesn’t request IO space."