Best Practices For Building A Multiple GPU System

Discuss anything you like on this forum.
Post Reply
philliplakis
Licensed Customer
Posts: 79
Joined: Wed Nov 19, 2014 2:27 am
Location: Sydney, Australia

smicha wrote:Tutor,
I am looking at photos of X10DRX and cannot find any info - is the 2011v3 socket a narrow ILM or a regular mount?

I have the DRG and its narrow.

Side note: No more cutting DVI's ... What do you think of the 1080ti's?

smicha wrote:Guys,

Do you know anything about pricing of the Klimax 210?
Also it looks like they are using the Supermicro board from the 4028GR-TRT2
Win10 - 6700k - 32gb Ram - GTX Titan X (Pascal) + GTX 1080
User avatar
smicha
Licensed Customer
Posts: 3151
Joined: Wed Sep 21, 2011 4:13 pm
Location: Warsaw, Poland

philliplakis wrote:
smicha wrote:Tutor,
I am looking at photos of X10DRX and cannot find any info - is the 2011v3 socket a narrow ILM or a regular mount?

I have the DRG and its narrow.

Side note: No more cutting DVI's ... What do you think of the 1080ti's?

smicha wrote:Guys,

Do you know anything about pricing of the Klimax 210?
Also it looks like they are using the Supermicro board from the 4028GR-TRT2
This is not Supermicro board - it has 20 pcie slots. This is their own construction.

As for 1080ti - this is a beast worth every single penny, at least for me.
3090, Titan, Quadro, Xeon Scalable Supermicro, 768GB RAM; Sketchup Pro, Classical Architecture.
Custom alloy powder coated laser cut cases, Autodesk metal-sheet 3D modelling.
build-log http://render.otoy.com/forum/viewtopic.php?f=9&t=42540
User avatar
Tutor
Licensed Customer
Posts: 531
Joined: Tue Nov 20, 2012 2:57 pm
Location: Suburb of Birmingham, AL - Home of the Birmingham Civil Rights Institute

smicha wrote:Tutor,
I am looking at photos of X10DRX and cannot find any info - is the 2011v3 socket a narrow ILM or a regular mount?
Although I don't have any X10DRXs, the pictures of the X10 that I've seen appear to support my suspicion that the 2011v3 sockets on the X10 are narrow width (side-to-side) like those on my X9s.

On a another note, my persistence in trying to get one of my X9DRX systems to boot with 20 GTX 1070s plus 2 GTX 980 Tis has garbled that X9's bios. So I have to take the system apart to restore the bios. I will certainly have to reduce the total GPU count.
smicha wrote:... . As for 1080ti - this is a beast worth every single penny, at least for me.

I fully agree. However, I hope that they aren't IO memory hogs as are my 780 and 980 Tis.
Because I have 180+ GPU processers in 16 tweaked/multiOS systems - Character limit prevents detailed stats.
User avatar
Notiusweb
Licensed Customer
Posts: 1285
Joined: Mon Nov 10, 2014 4:51 am

Tutor wrote:
On a another note, my persistence in trying to get one of my X9DRX systems to boot with 20 GTX 1070s plus 2 GTX 980 Tis has garbled that X9's bios. So I have to take the system apart to restore the bios. I will certainly have to reduce the total GPU count...


I fully agree. However, I hope that they aren't IO memory hogs as are my 780 and 980 Tis.

Hi Tutor,

1) Does X9DRX let you set the BCLK (Base Clock), I found that setting it higher or lower than default can impact performance and stability.
When 13 GPUs running full tilt on a render, it sometimes caused 1 or 2 to disconnect, so I have played with this in the past to try to enhance stability.
Default is usually 100 in my BIOS.
Note this is only as far as stability of multiple GPUs in system that are functional, it does not add IO space.
I found this relationship:
+BCLK = Higher Performance CPU, Lower stability GPU
-BCLK = Higher Stability GPU, lower performance CPU

2) Do some GPUs inherently use more or less IO space than others, or is it the connection schematic that uses more or less IO space (IE 1x vs 4x vs 8x vs 16x connection)
Win 10 Pro 64, Xeon E5-2687W v2 (8x 3.40GHz), G.Skill 64 GB DDR3-2400, ASRock X79 Extreme 11
Mobo: 1 Titan RTX, 1 Titan Xp
External: 6 Titan X Pascal, 2 GTX Titan X
Plugs: Enterprise
User avatar
Tutor
Licensed Customer
Posts: 531
Joined: Tue Nov 20, 2012 2:57 pm
Location: Suburb of Birmingham, AL - Home of the Birmingham Civil Rights Institute

Notiusweb wrote:....

Hi Tutor,

1) Does X9DRX let you set the BCLK (Base Clock), I found that setting it higher or lower than default can impact performance and stability.
When 13 GPUs running full tilt on a render, it sometimes caused 1 or 2 to disconnect, so I have played with this in the past to try to enhance stability.
Default is usually 100 in my BIOS.
Note this is only as far as stability of multiple GPUs in system that are functional, it does not add IO space.
I found this relationship:
+BCLK = Higher Performance CPU, Lower stability GPU
-BCLK = Higher Stability GPU, lower performance CPU

2) Do some GPUs inherently use more or less IO space than others, or is it the connection schematic that uses more or less IO space (IE 1x vs 4x vs 8x vs 16x connection)
Supermicro multi processor systems that allow slight CPU clock tweaking = the DAX line - ex. https://www.supermicro.com/products/mot ... DAX-iF.cfm. https://www.supermicro.com/manuals/moth ... L-1366.pdf .
X(9)(10)DRXs don’t provide that luxury.

Speed of data (I/O) movement determinants
x1 (example - Amfeltec Splitter one cable or http://i.ebayimg.com/images/g/f78AAOxy6 ... s-l500.jpg )
x4 (example - http://i.ebayimg.com/images/a/T2eC16h,! ... s-l500.jpg )
x8 (example - http://i.ebayimg.com/images/g/93MAAOxy4 ... s-l500.jpg )
x16 ( http://www.ebay.com/itm/NEW-PCIe-Expres ... 1117914385 and see also the bottom male PCI-e protrusion on most GPUs)
x16 is considered 16 times faster than x1; x8 is 8 times faster than x1, but only half as fast as x16 for speed of data movement between the GPU and CPU memory; so for us this mainly affects how fast does the GPU get projects and send back the render results, not the speed at which the GPU renders the project. CUDA core count and GPU speed settings are impactful for rendering speed, although also having faster "x" values would be of some benefit to animation rendering where multiple frame loads are needed.

Over and under clocking systems such as gaming or DAX systems that allow for such tweaks affects the PCIe transfer rate [ General Rule = Over clocking CPU usually = faster PCIe bus and under clocking usually = slower PCIe bus. But particular bios enhancements allow for hemi, demi, semi stages of detachments */]. The connection scheme (IE 1x vs 4x vs 8x vs 16x connection) generally has nothing directly to due with the IO space; but I can imagine situations where a system might otherwise be on the brink of instability and decreasing the speed of data throughput might save the day or increasing the speed of data throughput might push it over the edge.

2) Do some GPUs inherently use more or less IO space than others [?].

Yes, some GPUs, particularly, the latest ones, have more IO space needs. IO space is more like the “What is being required for the GPU and thus the system as a whole because the system must know the GPUs' identity and their resource needs and be able to satisfy those needs to boot cleanly and to run stably.” If the GPUs crave to much IO space the system becomes unstable. My Supermicro systems have bioses that will disable some GPUs to have enough resources available to tell the user/me the nature of the problem. Too much IO space consumption will muck up the system’s bios - which is what I recently caused - I didn't heed the warnings soon enough. IO space consumed by GPUs appears to be growing, particularly faster (especially for the GTX 780 TIs and 980 TIs) - in sum, GPUs with the TI designation. However, overall, as GPU cards become more complex/featurefull IO needs will likely increase for most of them.**/ And keep in mind that IO space consumption isn't limited to GPUs; so, getting a system to accommodate a certain number of GPUs might require disabling other non-GPU related features. But in the end, no motherboard can accommodate an infinite no. of GPUs - THERE IS A LIMIT FOR EVERY MOTHERBOARD MADE THUS FAR. I choose the Supermicro DRXs because their capacity (when combined with a Linux OS) appears to currently be the highest.

*/ Nehalem and Westmere CPU generations decoupled the CPU's speed from the PCIe bus speed - see, e.g., http://www.insanelymac.com/forum/topic/ ... ch-scores/. Intel ended this with SandyBridge and later GPUs but gave what I call "hemi, demi, semi stages of detachments" to some mainly gamer systems see, e.g., http://www.overclock.net/t/1198504/comp ... ck-edition . The net result was an end to extreme CPU overclocking.

**/ If one desires more precise information about IO space consumption, he/she should delve into the study of "bars," beginning here - https://en.wikipedia.org/wiki/PCI_configuration_space. Earlier in this thread I have referenced other sites with more detailed bar info. such as - http://resources.infosecinstitute.com/p ... nsion-rom/ . Those with a deeper interest should just begin by searching this thread for "io space." However, for those who don't want to dive this deeply, just remember to get a system with a bios that allows PCI-e configuration space to 4k, also sometimes called "above 4G" functionality. For those systems that have this functionality it's usually found in the PCI-e settings of the bios configuration options. Enable it before adding more than one GPU, by booting and accessing the bios options, then the PCI-e options and turning it on there. Don't buy a motherboard that lacks that functionality if you want to maximize GPU resources; so I recommend that you download the motherboard's manual and check for this functionality before you purchase..
Because I have 180+ GPU processers in 16 tweaked/multiOS systems - Character limit prevents detailed stats.
User avatar
smicha
Licensed Customer
Posts: 3151
Joined: Wed Sep 21, 2011 4:13 pm
Location: Warsaw, Poland

Tutor,

Does X9(10)DRX require any additional bios changes to run 11 GPUs in their all PCIe slots (no splitters, just one card per PCIe slot) ?
3090, Titan, Quadro, Xeon Scalable Supermicro, 768GB RAM; Sketchup Pro, Classical Architecture.
Custom alloy powder coated laser cut cases, Autodesk metal-sheet 3D modelling.
build-log http://render.otoy.com/forum/viewtopic.php?f=9&t=42540
User avatar
Tutor
Licensed Customer
Posts: 531
Joined: Tue Nov 20, 2012 2:57 pm
Location: Suburb of Birmingham, AL - Home of the Birmingham Civil Rights Institute

smicha wrote:Tutor,

Does X9(10)DRX require any additional bios changes to run 11 GPUs in their all PCIe slots (no splitters, just one card per PCIe slot) ?
Neither the X9DRX (see, e.g., https://www.superbiiz.com/detail.php?name=MB-X9DRXFB ) nor the X10DRX (see,e.g., https://www.newegg.com/Product/Product. ... 5EM2VH3311 or https://www.amazon.com/Supermicro-X10DR ... B00XQ3I14K ) should require any special bios tweaking (beyond possibly enabling above 4G functionality) to run 11 GPUs; however, I always recommend first updating the bios to the latest version (rarely have I received an X9 with the latest bios version) and enabling above 4G functionality.

Because, in the case of the X9, PCIe slots 8-11 (and in the case of the X10- slots 9-11) are blocked (after 7 in.) by a memory bank, you'd have to use cable risers for those slots or mini GPU cards like the GIGABYTE GeForce GTX 1070 Mini ITX OC GV-N1070IXOC-8GD Video Card [ see, e.g., https://www.newegg.com/Product/Product. ... 6814125893 or https://www.amazon.com/Gigabyte-GeForce ... B01JD2OSX0 and https://www.aliexpress.com/item/N-GV107 ... 2-VnYZvQVf ]. Of course since all of the DRX slots are x8 sized, you'd need to purchase those ~ $3.00 (USD) each low profile riser cards - see, e.g., http://www.ebay.com/itm/like/2010065667 ... noapp=true that I referenced in my last post. For each, now slightly raised, GPU card to be adequately secured to the rear of my Lian Li chassis, I purchased from my local hardware store 2 bags (six sets per bag) of #4-40 x 1" machine screws w/bolts for $1.00 per bag [ like these - http://www.homedepot.com/p/Crown-Bolt-4 ... /202707144 , but mine came with matching bolts ] ; leaving me with one extra set: 2 x 6 =12; 12-11 = 1. So in USD, I 'd spend ~$3 x11 = 33 + $2 (excluding tax), or ~ $35.00 extra if I were running minis, of coarse excluding the cost of the GPUs and the water blocks, etc.
Because I have 180+ GPU processers in 16 tweaked/multiOS systems - Character limit prevents detailed stats.
User avatar
smicha
Licensed Customer
Posts: 3151
Joined: Wed Sep 21, 2011 4:13 pm
Location: Warsaw, Poland

Tutor wrote:
smicha wrote:Tutor,

Does X9(10)DRX require any additional bios changes to run 11 GPUs in their all PCIe slots (no splitters, just one card per PCIe slot) ?
Neither the X9DRX (see, e.g., https://www.superbiiz.com/detail.php?name=MB-X9DRXFB ) nor the X10DRX (see,e.g., https://www.newegg.com/Product/Product. ... 5EM2VH3311 or https://www.amazon.com/Supermicro-X10DR ... B00XQ3I14K ) should require any special bios tweaking (beyond possibly enabling above 4G functionality) to run 11 GPUs; however, I always recommend first updating the bios to the latest version (rarely have I received an X9 with the latest bios version) and enabling above 4G functionality.

Because, in the case of the X9, PCIe slots 8-11 (and in the case of the X10- slots 9-11) are blocked (after 7 in.) by a memory bank, you'd have to use cable risers for those slots or mini GPU cards like the GIGABYTE GeForce GTX 1070 Mini ITX OC GV-N1070IXOC-8GD Video Card [ see, e.g., https://www.newegg.com/Product/Product. ... 6814125893 or https://www.amazon.com/Gigabyte-GeForce ... B01JD2OSX0 and https://www.aliexpress.com/item/N-GV107 ... 2-VnYZvQVf ]. Of course since all of the DRX slots are x8 sized, you'd need to purchase those ~ $3.00 (USD) each low profile riser cards - see, e.g., http://www.ebay.com/itm/like/2010065667 ... noapp=true that I referenced in my last post. For each, now slightly raised, GPU card to be adequately secured to the rear of my Lian Li chassis, I purchased from my local hardware store 2 bags (six sets per bag) of #4-40 x 1" machine screws w/bolts for $1.00 per bag [ like these - http://www.homedepot.com/p/Crown-Bolt-4 ... /202707144 , but mine came with matching bolts ] ; leaving me with one extra set: 2 x 6 =12; 12-11 = 1. So in USD, I 'd spend ~$3 x11 = 33 + $2 (excluding tax), or ~ $35.00 extra if I were running minis, of coarse excluding the cost of the GPUs and the water blocks, etc.

I'll be cutting RAM slot clippers on lowest DDR slots and opening closed x8 PCI ports... :mrgreen:
3090, Titan, Quadro, Xeon Scalable Supermicro, 768GB RAM; Sketchup Pro, Classical Architecture.
Custom alloy powder coated laser cut cases, Autodesk metal-sheet 3D modelling.
build-log http://render.otoy.com/forum/viewtopic.php?f=9&t=42540
User avatar
Tutor
Licensed Customer
Posts: 531
Joined: Tue Nov 20, 2012 2:57 pm
Location: Suburb of Birmingham, AL - Home of the Birmingham Civil Rights Institute

smicha wrote: I'll be cutting RAM slot clippers on lowest DDR slots and opening closed x8 PCI ports... :mrgreen:

I've also opened a few of mine. N.B. : The front CPU also blocks slots 10 & 11 when using full length cards.
Last edited by Tutor on Sun Mar 05, 2017 5:54 pm, edited 1 time in total.
Because I have 180+ GPU processers in 16 tweaked/multiOS systems - Character limit prevents detailed stats.
User avatar
smicha
Licensed Customer
Posts: 3151
Joined: Wed Sep 21, 2011 4:13 pm
Location: Warsaw, Poland

Tutor wrote:
smicha wrote: I'll be cutting RAM slot clippers on lowest DDR slots and opening closed x8 PCI ports... :mrgreen:



N.B. the front CPU also blocks slots 10 & 11.
I think it is close to 11th but not sure... look at the attached image
You do not have the required permissions to view the files attached to this post.
3090, Titan, Quadro, Xeon Scalable Supermicro, 768GB RAM; Sketchup Pro, Classical Architecture.
Custom alloy powder coated laser cut cases, Autodesk metal-sheet 3D modelling.
build-log http://render.otoy.com/forum/viewtopic.php?f=9&t=42540
Post Reply

Return to “Off Topic Forum”