Best Practices For Building A Multiple GPU System

Discuss anything you like on this forum.
Post Reply
User avatar
Notiusweb
Licensed Customer
Posts: 1285
Joined: Mon Nov 10, 2014 4:51 am

Tutor, do you ever run your 12 Titan Z on bench, do you get the same score as me? Or higher?... :mrgreen:
If they create a V3 bench you will beat me every time with your 6 Zs, and the rest (Professor and MaryAnn....) :?
As will others with networked rigs.... :cry:
Until I get a 2nd rig up and running, networked... :twisted:...it will be aircooled :lol:
Actually, would a rig + networked rig count for one benchscore? :roll:
Wait...17 OS GPU rendering rigs?...DAAAAAAAAAAAAAAAAAAAAAAAAAMMMMMMMMMMMMMMMMMMNNNNNNNNNNNNNNNN :shock:
What's this Smily for :arrow:
Win 10 Pro 64, Xeon E5-2687W v2 (8x 3.40GHz), G.Skill 64 GB DDR3-2400, ASRock X79 Extreme 11
Mobo: 1 Titan RTX, 1 Titan Xp
External: 6 Titan X Pascal, 2 GTX Titan X
Plugs: Enterprise
User avatar
smicha
Licensed Customer
Posts: 3151
Joined: Wed Sep 21, 2011 4:13 pm
Location: Warsaw, Poland

Tutor,

'Thank you' is not enough for your reply ;)

I want to use X10DRX and split all 10 PCIe slots that are x8 (#1 is x4, #2-3 are closed, #4-11 are open at their bottom (or ends depending how you look at them, but I bet you know this) so that they can be used for 8 watercooled single slotted gpus with x16 slots). Currently I am looking for a splitter to get 2 x16 slots from each of x8 slots

http://www.aliexpress.com/item/Free-Tra ... 9.3.JdIBh7
but I have doubts they will give me two slots at x4 - what do you think?

or supermicro RSC-R2UT-2E8R - I am not sure if there is a room for cables there when I will use them for all 10 slots that are 8x
http://www.supermicro.com/ResourceApps/Riser.aspx

My custom designed case is similar to CaseLabs S8 but uses more 'pedestals' to fit all 20 gpus. I'll come up with it soon here on the forum.

So little bit sad that Windows sees max of 14 gpus. Have you tested windows 10 to see 20 of them?

EDIT: just found this
http://www.supermicro.com/a_images/prod ... 2E16-A.jpg

so the solution for 20 gpus - use 10 of the above supermicro risers and 10 riser cables that go to x8 slots (for the first two x8 slots that are closed use x8 -> x16 riser cables, or cut the plastic part of x8 slots to make them open - make them exactly the same as #4-11 slots)
Last edited by smicha on Sat May 28, 2016 10:15 am, edited 1 time in total.
3090, Titan, Quadro, Xeon Scalable Supermicro, 768GB RAM; Sketchup Pro, Classical Architecture.
Custom alloy powder coated laser cut cases, Autodesk metal-sheet 3D modelling.
build-log http://render.otoy.com/forum/viewtopic.php?f=9&t=42540
User avatar
itou31
Licensed Customer
Posts: 377
Joined: Tue Jan 22, 2013 8:43 am

Hi Tutor,

Need your advice also :
I will get a X8DTH-IF with 2 X5570 for a spare rig. the slot are 8x speed but in 16x format. I think to put 6 GPU later on the board and add 4 other on the amfeltec splitter on the last slot. The BIOS seems to not have 4G decoding. Do you think that I will get out of ressources ? with my main rig (a 3930K in asus P9X79) I can manage only 9 GPUs.
Thanks
I7-3930K 64Go RAM Win8.1pro , main 3 titans + 780Ti
Xeon 2696V3 64Go RAM Win8.1/win10/win7, 2x 1080Ti + 3x 980Ti + 2x Titan Black
User avatar
Tutor
Licensed Customer
Posts: 531
Joined: Tue Nov 20, 2012 2:57 pm
Location: Suburb of Birmingham, AL - Home of the Birmingham Civil Rights Institute

If A Mind Is A Terrible Thing To Waste, Then A Warped Mind Is The Most Terrible Thing To Waste
Notiusweb wrote:Tutor, do you ever run your 12 Titan Z on bench, do you get the same score as me? Or higher?... :mrgreen:
Yes, I've benched my 12 Titan Z GPU cores. No, I didn't get the same score as you. Yes, it was higher, more likely in large part because of this mind warping technique - download/file.php?id=44935&mode=view - to roll mozzarella sandwiches. Remember our discussion about mozzarella sandwiches in July 2015 - viewtopic.php?f=40&t=43597&start=210#p242963 . If you ever develop a taste for mozzarella sandwiches and want to know, in detail, how I make mine, I'm sure that you know that all you have to do is ask me. In honor of a poster who implied late last year that the highest scores could only be achieved during the winter, I'm waiting for this summer to re-test those Zs. I will then have had an additional year of mind warping.

I fully recognize that I have a warped mind and I also recognize that a warped mind shouldn't be wasted - viewtopic.php?f=40&t=43597&start=220#p243126 . So as not to waste my warped mind, I try to feed it throughout each and every day by consuming mind warping information to warp my mind even further. Daily, I try to warp my reality and that of those who care to join me in warped realities. If my brain brain is ever physically examined, I want it to be the most wrinkled brain ever seen, evidencing massive warping.
Notiusweb wrote:... .Actually, would a rig + networked rig count for one benchscore? :roll:
That's a new one on me since I don't use Octane's network rendering feature and I haven't otherwise tried running OctaneBench with more than one of my systems at a time.
Notiusweb wrote:... .What's this Smily for :arrow:
I haven't yet gotten much into gifing/smilies, but if the pic were meant to be truly informative as a word substitute, I'd guess that it meant "Look here or Follow this."
Because I have 180+ GPU processers in 16 tweaked/multiOS systems - Character limit prevents detailed stats.
User avatar
Tutor
Licensed Customer
Posts: 531
Joined: Tue Nov 20, 2012 2:57 pm
Location: Suburb of Birmingham, AL - Home of the Birmingham Civil Rights Institute

Here’s inspiration for your quest, especially it’s intro: Funkadelic - One Nation [or as my warped mind would now begin it - “20 GPUs] Under A Groove (12" Mix) (Slayd5000) , - https://www.youtube.com/watch?v=wjbVmX1_wh8 - */, but you might also profit from listening to the rest of it.
smicha wrote:Tutor,

'Thank you' is not enough for your reply ;)

I want to use X10DRX and split all 10 PCIe slots that are x8 (#1 is x4, #2-3 are closed, #4-11 are open at their bottom (or ends depending how you look at them, but I bet you know this) so that they can be used for 8 watercooled single slotted gpus with x16 slots). Currently I am looking for a splitter to get 2 x16 slots from each of x8 slots

http://www.aliexpress.com/item/Free-Tra ... 9.3.JdIBh7
but I have doubts they will give me two slots at x4 - what do you think?
As your edit reveals, I’m glad that you’ve chosen a different riser route. That Aliexpress product connects modern PCIe slots to older technology PCI devices, which I strongly believe is not your goal.
smicha wrote: … or supermicro RSC-R2UT-2E8R - I am not sure if there is a room for cables there when I will use them for all 10 slots that are 8x
http://www.supermicro.com/ResourceApps/Riser.aspx

My custom designed case is similar to CaseLabs S8 but uses more 'pedestals' to fit all 20 gpus. I'll come up with it soon here on the forum.

So little bit sad that Windows sees max of 14 gpus. Have you tested windows 10 to see 20 of them?
No, I haven’t used and do not now intend to ever use Windows 10.
smicha wrote: EDIT: just found this
http://www.supermicro.com/a_images/prod ... 2E16-A.jpg

so the solution for 20 gpus - use 10 of the above supermicro risers and 10 riser cables that go to x8 slots (for the first two x8 slots that are closed use x8 -> x16 riser cables, or cut the plastic part of x8 slots to make them open - make them exactly the same as #4-11 slots)
Here’s my take on some of the constrictions:

The GTX GPUs that we want to use are thick, tall, long and generate heat. Their thickness and heat generating ability means that they have to be spaced properly. Even with water cooling, generally they can’t be spaced any closer than they could be space if inserted directly into any other similarly configured PCIe slotted motherboards that I’ve ever seen. If they are arrayed sideways as the RSC-R2UG-A2E16-A pic indicates that you have in mind, how will each dual grouping be arrayed so that you won’t soon run out of riser cable length? What riser cable lengths will you be using? I recognize that the riser boards will give you a little more leeway, but how much more? Given my GPUs' height and thickness, I had to place some of them on the motherboard just to have room to place others on risers and not overrun the risers' length. Moreover, I'm using dual GPU Titan Zs, so I have fewer than 20 (i.e.,14) cards. A drawing might help me to see better what you have in mind to dance your way out of the endeavor's constrictions.

P.S. “20 GPUs, don’t fail me now. And if you don’t fail me now, who can stop us?” Lyrics: http://www.metrolyrics.com/one-nation-u ... delic.html

*/ My mind is old-school warped.
Because I have 180+ GPU processers in 16 tweaked/multiOS systems - Character limit prevents detailed stats.
User avatar
Tutor
Licensed Customer
Posts: 531
Joined: Tue Nov 20, 2012 2:57 pm
Location: Suburb of Birmingham, AL - Home of the Birmingham Civil Rights Institute

itou31 wrote:Hi Tutor,

Need your advice also :
I will get a X8DTH-IF with 2 X5570 for a spare rig. the slot are 8x speed but in 16x format. I think to put 6 GPU later on the board and add 4 other on the amfeltec splitter on the last slot. The BIOS seems to not have 4G decoding. Do you think that I will get out of ressources ? with my main rig (a 3930K in asus P9X79) I can manage only 9 GPUs.
Thanks
What OS do you intend to run on it? The response in the next paragraph assumes the answer = Windows. If the answer = Linux, then the response would be that getting 9-10 GPUs working wouldn't be at all unlikely. Under Linux, I've gotten 16 GPU processors (8 x GTX 590s) working on my Westmere supported Tyan, but it's manufacturer intended GPU processor limit is 8 and under Windows the system with that same load wouldn't boot completely. Moreover, I just stopped at 16 GPU processors. It might be able to go higher. Under Windows, 9 GPU processors (originally I had thought it was 8 because I had been using GTX 590s and Titan Zs for testing) is the max. for that system. Always keep in mind that different GPU models have different IO space requirements and that, as a general rule and trend, the newer the GPU family, the more IO space they need [they tend to have more functionality to account for than do their predecessors]. Thus, factors such as these also affect the cap.

Few Nehalem/Westmere (i.e., for Intel 5500/5600 series CPU) motherboards have an above 4G decoding selection. GPU computing wasn’t very prevalent back then. Thus, one’s rolling dice/speculating whether someone can get 2 or 3 or 4 or more GPUs [beyond the number of PCIe slots] to work in such systems (usually adding 1 additional GPU [beyond the number of PCIe slots] is possible on almost any Nehalem/Westmere supported motherboard regardless of OS). The one thing that you’d have in your favor is that Supermicro systems generally have very well developed bioses and it’s been my experience that Supermicro will make the effort to optimize the bios further upon customer request. However, the board you reference is sort of ancient now and I’m guessing that you’re not the first owner. Thus, unless it’s a gift or a crazily low priced deal, I recommend that you explore motherboards that support Intel’s next gen or higher processors (Sandy/then Ivy bridge/then Haswell/then Broadwel/then Skylake - latest)*/ and consider searching for suitable CPUs on Ebay, where I’ve found great bargains. That’s what I'd do particularly if I wanted to install 10 or more GPUs processor in a system wit minimal hassle.

*/ I routinely search Intel’s site - http://ark.intel.com - where I can read the specs for any CPUs that I’m interest in, looking particularly for (1) PCI Express Rev., (2) Max # of PCI Express Lanes supported, (3) Max CPU Config., (4) Memory Spec., (5) Core #/Threads # and of course speeds/frequencies. Also, I always download and read manuals (paying particular attention to board pics and schematics) before any motherboard purchase, looking particularly for, among others, “above 4g” functionality and slot count (& if >1 CPU - lane sourcing), slot blockers (like unsound memory slot placement) and lane type.

P.S.
Irony #1. My having similar concerns/issues with a pre-Sandybridge motherboard is why this thread was created. So, I understand your concerns.
Irony #2. Those concerns/issues have further led to me warping my mind more than I could ever have originally guessed.
Because I have 180+ GPU processers in 16 tweaked/multiOS systems - Character limit prevents detailed stats.
User avatar
smicha
Licensed Customer
Posts: 3151
Joined: Wed Sep 21, 2011 4:13 pm
Location: Warsaw, Poland

Tutor,

What Linux are you using? I am a complete rookie in this topic. Few years ago I tried to get started with Ubuntu but gave it up quickly.
3090, Titan, Quadro, Xeon Scalable Supermicro, 768GB RAM; Sketchup Pro, Classical Architecture.
Custom alloy powder coated laser cut cases, Autodesk metal-sheet 3D modelling.
build-log http://render.otoy.com/forum/viewtopic.php?f=9&t=42540
User avatar
itou31
Licensed Customer
Posts: 377
Joined: Tue Jan 22, 2013 8:43 am

Thanks Tutor for the description.
I ckecked the X8DTH-IF that have 2 controller 5520 chipset, so up to 72 PCIe lanes (40 PCIe lanes on my P9X79 3930K). So the board is not bad !
I could have the complete rig (used) X8DTH-IF + 2 X5570 + 48Go RAM ECC for 350€. A good price I think ?
I7-3930K 64Go RAM Win8.1pro , main 3 titans + 780Ti
Xeon 2696V3 64Go RAM Win8.1/win10/win7, 2x 1080Ti + 3x 980Ti + 2x Titan Black
User avatar
Notiusweb
Licensed Customer
Posts: 1285
Joined: Mon Nov 10, 2014 4:51 am

Tutor, can the supermicro boards run Windows 7, 8, 10 fine, or are there certain compatibility issues that are known?
PS - how does it feel to have to parent us crazy kids :D
Win 10 Pro 64, Xeon E5-2687W v2 (8x 3.40GHz), G.Skill 64 GB DDR3-2400, ASRock X79 Extreme 11
Mobo: 1 Titan RTX, 1 Titan Xp
External: 6 Titan X Pascal, 2 GTX Titan X
Plugs: Enterprise
User avatar
Tutor
Licensed Customer
Posts: 531
Joined: Tue Nov 20, 2012 2:57 pm
Location: Suburb of Birmingham, AL - Home of the Birmingham Civil Rights Institute

smicha wrote:Tutor,

What Linux are you using? I am a complete rookie in this topic. Few years ago I tried to get started with Ubuntu but gave it up quickly.
I'm currently running Linux Mint v17.2 ( which uses Ubuntu’s software repositories - http://www.howtogeek.com/191207/10-of-t ... -compared/ ) and started running Mint when it was at v13. Previously, I had used Linux OpenSuse.
Because I have 180+ GPU processers in 16 tweaked/multiOS systems - Character limit prevents detailed stats.
Post Reply

Return to “Off Topic Forum”