Best Practices For Building A Multiple GPU System

[smicha]

$4k is way too much, but if the trenton backplane and the dual xeon boards cost about 800$ that would make a very attractive option.

What I am thinking, Tutor, is watercooling 12 or even more gpus at once (8 in the lower chamber of TX10, 4 in one of its sides, or even 4 more in another side as a separate system). 2 or 3 x 2000W superflowers would power them up.

http://www.caselabs-store.com/magnum-tx10/

[glimpse]

Tutor, think I've seen at least one Guy already on this forum who utilise one of Trenton backplates & it worked (though he wasn't pushing with max on GPUs =)
Curious to see if that would work for You - might be interested to purchace one too as latelly got some Guys asking to build "slightly" more powerfull mosnters.

[Tutor]

From https://bitcointalk.org/index.php?topic=64450.0

Trenton BPX6806 backplane + $600.00
Trenton NLT6313 host board + $210.00


So I assume that overall cost (mobo + backplane) shall not exceed $1k.

Assumed prices much too low, unless you catch someone in the act of fire-selling a used part. That Single Board Computer - the Trenton NLT6313 host board (that’s the drop in CPU/ram/peripheral connection card) that you listed for around $210 - is now legacy (i.e., not being sold by Trenton any longer) and was sold by Trenton for a lot more when it was then current.

First, only certain, applicable CPUs listed in Trenton’s literature have been qualified to work with the various systems. Secondly, these quoted prices do not include any case, PSU, storage, cables, etc., unless they're specifically listed as an included, not includable, item in the listing. All prices are in USD.

The most modern Single Board Computer - the Trenton HEP8225 HDEC Series System Host Board [ http://www.trentonsystems.com/single-bo ... host-board ], with two E5-2680 V3s, has a quoted price of $5,971.00 (The Single Board Computer isn’t sold without any CPUs and that price doesn't include ram). The 8 slotted Backplane - the HDB8228 [ http://www.trentonsystems.com/backplane ... -backplane ] - has a quoted price of $932.00.

The Single Board Computer - the Trenton JXT6966 [ http://www.trentonsystems.com/single-bo ... d-computer ], with two low-powered LGA 1366 CPUs, i.e, circa 2009 certain EC or LC 5500s, has a quoted price of around $3,000.00 (The Single Board Computer isn’t sold without any CPUs and that price doesn't include ram). The BPX6806 18-slot PCI Express BackPlane has a quoted price of $1,243.00. It's the oldest of their two massively slotted backplanes; and this Single Board Computer - the Trenton JXT6966 - is just a little ancient. They recommended that to save some cash, use that BackPlane with the TSB7053 Single Board Computer [ http://www.trentonsystems.com/single-bo ... d-computer ] (single CPU processor) that has a base price w/the lowest cost CPU (The Single Board Computer isn’t sold without any CPUs) of about $1,400 without ram. It costs an additional $472.00 (4x8GB sticks @ $118 ea.) with 32 gigs of ram (that’ll “Max it out.”).

The Single Board Computer - the Trenton BXT7059 [ http://www.trentonsystems.com/single-bo ... d-computer ], with two E5-2448s, has a quoted price of $4,407.00. The BPG8032 PCI Express BackPlane [ http://www.trentonsystems.com/backplane ... -backplane ] with 18-slots, but called, "16-slot" because of its too close slot proximity to the Single Board Computer, has a quoted price of $3,291.00. Its the one specifically mentioned in Trenton’s literature as being suited for "GPU computing solutions" (but they did confirm that the BPX6806 should also be fine for CUDA computing, but not the best for high end video production since it’s limited to PCIe 2.0).

Tutor, think I've seen at least one Guy already on this forum who utilise one of Trenton backplates & it worked (though he wasn't pushing with max on GPUs =)
Curious to see if that would work for You - might be interested to purchace one too as latelly got some Guys asking to build "slightly" more powerfull mosnters.

Something ( [ [ https://devtalk.nvidia.com/default/topi ... -it-works/ ] + http://www.superbiiz.com/detail.php?name=MB-X9DRXF# ] + [ http://www.supermicro.com/products/chas ... -R1620.cfm ] + [ http://www.superbiiz.com/detail.php?name=CA-747TQ6B ] + [ I've long had four homeless E5-4650s V1 ESs that my Supermicros see as E5-2680s (just slightly faster ones because of an extra turboboost bin) when installed as pairs (and not as quads) and I have some really fast ram and storage that I've yet to find a use for] = < $$$ leaving my pockets) tells me that I'll remain SuperMicroMan. Now, if someone would just give to me as a gift four Titan X Hydros, then that would complete the aquatic Titan family, which now consists of six Titan Zs Hydros (12 GPUs), one Titan Hydro and one Titan Black Hydro (4 + 12 + 1 + 1 = 18); thus, allowing me to render quickly using Octane and Redshift ("Unlimited number of GPUs - but have to run additional instances where no. is > 8") or Thea ("Unlimited number of GPUs") and/or FurryBall RT ("Unlimited number of GPUs") simultaneously. Would have preferred inviting only Octane to this party, but I'd get singled out because this house party's count will exceed 12, even tho' reasonable minds should say that I currently have permission to invite, at least, 84 (7x12) Octane guests. Some mates are so narrow-minded and stupid that they impose limitations that induce flirtatious behavior. As our houses grow, flirtation with others might become the order of the day and rampant.

P.S. The wonderful Trenton rep who serviced my call resides and works in my home state - just about an hour's drive away.

[smicha]

Oh, this pricing is crazy. 3200$ for the backplane? And the single board is sold with xeons only? $9k for both? I am very disappointed.

Tutor,

Thank you very much for such thorough information.

[glimpse]

Thanks for all the great info Tutor! =) serious stuff out there.. but in the end all this long thread brings me to some sort of conclusion, that best performance for the buck lies leveraging consumer (prosumer) grade gaming/workstation oriented motherboard with 40Lane CPU + PLX (or pair of those) - that brings us nearlly to designed ceiling of those rigs. Having 4X dual GPUs or 7-8 single GPU cards on raisers or moded into single slot to fit directly on the board - all watercooled is the best deal You can get. Even a single step further explodes price too much, knocking down value - if You're usign rig for GPU workloads spending all the extra money for features You don't need is not so wise in the end.

[Tutor]

Dup of earlier version of post below.

[Tutor]

Oh, this pricing is crazy. 3200$ for the backplane? And the single board is sold with xeons only? $9k for both? I am very disappointed.

Tutor,

Thank you very much for such thorough information.

Yes - $3,200 for one piece of a two piece motherboard is crazy. There's a better alternative, however. I'll be a bit more thorough ->

Thanks for all the great info Tutor! =) serious stuff out there.. but in the end all this long thread brings me to some sort of conclusion, that best performance for the buck lies leveraging consumer (prosumer) grade gaming/workstation oriented motherboard with 40Lane CPU + PLX (or pair of those) - that brings us nearlly to designed ceiling of those rigs. Having 4X dual GPUs or 7-8 single GPU cards on raisers or moded into single slot to fit directly on the board - all watercooled is the best deal You can get. Even a single step further explodes price too much, knocking down value - if You're usign rig for GPU workloads spending all the extra money for features You don't need is not so wise in the end.

On my present Supermicro systems, I intend to use a combo of risers and splitters. Since I'm also leaning towards what I've discussed most recently above and will further discuss a little more below, I'll probably also use PCI Express Riser Card x8 to x16 Left Slot Adapters, along with risers and splitters for my Supermicro X9DRX build.

(Prices are in USD and do not include taxes, any import fees or shipping)
A few more things to keep in mind. Supermicro and Tyan have not been traditionally known as general consumer (or even prosumer) motherboard/system manufactures - their claim to fame was their servers and workstations. General consumer motherboard/system manufactures, including but not limited to Asus [ http://www.asus.com/us/Commercial_Servers_Workstations/ ] and Gigabyte [ http://b2b.gigabyte.com ] began reaching up into the server space starting a few years ago, particularly when the LGA 2011s were released (EVGA did it a couple of times with the dual Xeon EVGA SR-2 in 2010 for the LGA 1366 and the dual Xeon EVGA SR-X for the LGA 2011s, but didn't achieve much traction and quit). Supermicro appears to have welcomed the challenge by starting to reach down by taking their military grade know -how into the enthusiast and general consumer space at general consumer and enthusiast price points (See, e.g., http://www.newegg.com/Product/Product.a ... 6813182718 ]. When you get some free time, take a look at all of the general consumer-focused motherboards now being Supermicro branded [ See, e.g., http://www.supermicro.com/products/nfo/gaming.cfm ]. Here's what Supermicro has achieved in the DayTrader, GPU/CPU enthusiast and prosumer space:
1) For LGA 2011 V.1, V.2 and V.3, the best dual CPU over-clocking motherboards are Supermicro's DAX line (for under $700, $600 and $500, depending on the feature set [ http://www.superbiiz.com/query.php?s=supermicro+dax ]; the EVGA SR-2's retailed for around the $625 point - Who in 2010 who have thought that Supermicro would be selling the dominant overclocking hardware and at such prices?
2) The only motherboard that we know for sure has reached, and has passed, the 12 GPU threshold {recognized and working in CUDA}, is the Supermicro X9DRX line [ see https://devtalk.nvidia.com/default/topi ... -it-works/ and http://www.superbiiz.com/query.php?s=Supermicro+X9DRX ]. I firmly believe that we can get this platform to work in Octane to reach the 12 GPU license limit, if not with Windows, then with either Windows Server or with Linux (which, of course, is free). ZZZ1000 used Linux [ https://devtalk.nvidia.com/default/topi ... s/#4030904 ]. He first got 16 GPUs running under CUDA 5.5 in Linux and he later reached 18 GPUs running under CUDA in Linux. The only step that I can't tell whether he took was getting all of his GPUs to run under Octane. BTW - Octane has Linux support for Maya, Blender, Modo and Nuke. You can use Wine [ https://www.winehq.org/download/ ] to run other 3d applications under Linux such as Cinema 4d [ http://www.c4dcafe.com/ipb/forums/topic ... -on-linux/ ] etc. [ http://www.linux.com/learn/answers/view ... s-in-linux ]. Moreover, "The SuperMicro X9DRX+-F, a favourite amongst the rendering crowd... ." [ http://www.techspot.com/community/topic ... us.202086/ ]

Here's my slight twist to your assessment:
The best performance for the buck for those desiring to run 12 GPUs on Octane in one system currently lies in our leveraging low cost Supermicros - they're more likely to bring us, for the least cost, to the current Octane license ceiling for those rigs. The Supermicro X9DRX line is cheap (in terms of price), particularly when compared to the Trenton systems. Even when compared to the prices of some consumer (prosumer) grade gaming/workstation oriented motherboards, the Supermicros aren't but minimally more expensive, and the Supermicros aren't more expensive than every consumer (prosumer) grade gaming/workstation. One can purchase an 11 slotted Supermicro X9DRX in two versions, one for $460 [ http://www.superbiiz.com/detail.php?name=MB-X9DRXFB ] */ and the other for $489 [ http://www.superbiiz.com/detail.php?name=MB-X9DRXF ] **/ . One can purchase a Supermicro upright chassis that houses either motherboard for $967 [ http://www.superbiiz.com/detail.php?name=CA-747TQ6B ] ***/ { I recently purchased two EVGA 1600W PSUs that together cost me $640 { $967 - $640 = $327; however, the two 1620W Supermicro PSUs are operating in a power redundancy capacity - but for a value comparison, I think that the point is clear}. That chassis also includes the ability to use the GPUs, storage and memory of your choice (so long as it meets LGA 2011 standards). Purchasing the more expensive motherboard and chassis separately costs about $1456, or the least expensive motherboard and the same chassis for $1,427 ($967 + $460 = $1,427). But if one doesn't want to bother with installing the motherboard onto the chassis, one can purchase a motherboard/chassis combo for $1,552 [ http://www.superbiiz.com/detail.php?name=SY-747RTXF ] ****/; the combo comes with the lesser expensive motherboard. You can also buy a rackmount chassis combo that has the least expensive motherboard for $1,346.99 [ http://www.superbiiz.com/detail.php?name=SY-637RTXF ]. It has 980W Redundant Power Supplies. If you’re going to be using lots of externally powered splitters this would likely work, but if you’re going to be using only or mainly risers, then connecting up to eleven GPUs to the eleven slots may be too much, since each GPU may draw up to 75 watts from its connected PCIe slot: 75W x 11 = 825W, not including the power requirements of the CPUs, chip set, fans, ram, etc. In any case, however, going with Supermicro is a much less expensive route to pursue than going the Trenton route. But I do admit one potential downside of Supermicro's chassis, depending on one's tastes Supermicro computers aren't flashy looking. However, I've never had any complaint about the quality of any Supermicro product.

Differences Bolded
*/
Specifications
Mfr Part Number: X9DRX+-F-B
CPU: Dual LGA2011
Support Intel Xeon E5-2600 Processors
Cache up to 20MB
QPI up to 8 GT/s
Support up to 135W TDP
Chipset: Intel C602
Memory: 16x 240pin DDR3-1600/1333/1066/800 DIMM Slots, Supports up to 512GB ECC/REG Memory or up to 128GB ECC/Non-ECC Unbuffered Memory
Slots: 10x PCI-Express 3.0 x8 Slots, 1x PCI-Express 2.0 x8 Slot (runs at x4)
SATA: 2x SATA3 Ports (via AHCI), 8x SATA2 Ports (four via AHCI, four via SCU); Support RAID 0, 1, 5, 10 [see http://www.tomsitpro.com/articles/xeon- ... 288-3.html - essentially, it adds SAS functionality ]
Video: Matrox G200eW Graphics Controller
LAN: Intel i350 Dual Port Gigabit Ethernet Controller; 1x Realtek RTL8201N PHY (Dedicated IPMI)
Ports: 10x USB 2.0 Ports (4 rear, 4 via headers, 2 Type A), 1x VGA Port, 2x Serial Ports (1 rear, 1 via header), 2x RJ45 LAN Ports, 1x RJ45 Dedicated IPMI LAN Port
Form Factor: Proprietary, 15.2 x 13.2 inch / 38.6 x 33.5 cm
RoHS Compliant

**/
Specifications
Mfr Part Number: X9DRX+-F-O
CPU: Dual LGA2011
Supports Intel Xeon E5-2600 Processors
Cache up to 20MB
QPI up to 8 GT/s
Supports up to 135W TDP
Chipset: Intel C602
Memory: 16x 240pin DDR3-1600/1333/1066/800 DIMM Sockets, Supports up to 512GB ECC/REG Memory or up to 128GB ECC/Non-ECC, Unbuffered Memory
Slots: 10x PCI-Express 3.0 x8 Slots, 1x PCI-Express 2.0 x8 Slot (runs at x4)
SATA: 2x SATA3 Ports, 8x SATA2 Ports; Support RAID 0, 1, 5, 10
Video: Matrox G200eW Graphics Controller
LAN: Intel i350 Dual Port Gigabit Ethernet Controller; 1x Realtek RTL8201N PHY (dedicated IPMI)
Ports: 10x USB 2.0 Ports (4 rear, 4 via headers, 2 Type A), 1x VGA Port, 2x Serial Ports (1 rear, 1 via header), 2x RJ45 LAN Ports, 1x RJ45 Dedicated IPMI LAN Port
Form Factor: Proprietary, 15.2 x 13.2 inch / 38.6 x 33.5 cm
RoHS Compliant


***/
Specifications
Mfr Part Number: CSE-747TQ-R1620B
Case Type: 4U Rackmount / Tower
Color: Dark Gray
M/B Type: EATX, ATX; Max Motherboard Size - 15.2 x 13.2 inch
Processor Support: Dual, Single Intel/AMD Processors
Drive Bays: 8x 3.5" Hot-swap SAS/SATA Drive Bays, 3x 5.25" Peripheral Drive Bays in Sorage Module (Rotates 90° for Rackmount), 1x 3.5" Fixed Peripheral Drive Bay
Expansion Slots: Capable of housing 11x Full-Height, Full-Length Expansion Cards
Backplane: SAS/SATA Hard Drive Backplane w/ SES2
Cooling System: 4x (92x38mm) 4-pin PWM Middle Cooling Fans, 2x (80x38mm) 4-pin PWM Rear Exhaust Fans
Front Panel:
Buttons: LED Power On/Off, System Reset
LEDs: Power Status, Hard Drive Activity, 2x Network Activity, System Overheat & Power Fail
Ports: 2x USB 2.0 Ports
Power Supply: 1620W 80 PLUS Platinum Certified High-efficiency Redundant Power Supply w/ PMBus
Dimensions (WxDxH): 18.2 x 26.5 x 7.0 inch / 462.0 x 673.0 x 178.0 mm
Gross Weight: 62.0 lbs / 28.1 kg

****/
Specifications
Mfr Part Number: SYS-7047R-TXRF
Motherboard: Super X9DRX+-F
Processor: Dual LGA2011
Supports Intel Xeon E5-2600 Processors
Cache up to 20MB
QPI up to 8 GT/s
Chipset: Intel C602
Memory: 16x 240pin DDR3-1600/1333/1066/800 DIMM Slots, Supports up to 512GB ECC/REG Memory or up to 128GB ECC/Unbuffered Memory
Slots: 10x PCI-Express 3.0 x8 Slots; 1x PCI-Express 2.0 x8 Slot (runs at x4)
SATA: 2x SATA3 Ports, 8x SATA2 Ports, Supports RAID 0, 1, 5, 10
Video: Matrox G200eW Graphics
LAN: Intel i350 Dual Port Gigabit Ethernet Controller; 1x Realtek RTL8201N PHY (dedicated IPMI)
Drive Bays: 8x Hot-swap 3.5" Drive Trays; 3x 5.25" Peripheral Drive Trays
Backplane: SAS/SATA Hard Drive Backplane w/AMI MG9072
Ports: 10x USB 2.0 Ports (4 rear, 4 via header, 2 Type A), 2x Serial Ports (1 rear, 1 via header), 1x VGA Port, 2x RJ45 LAN Ports, 1x RJ45 Dedicated IPMI LAN Port
Form Factor: 4U Rackmountable / Tower
Power Supply: 1280W 80 Plus Platinum Certified Redundant High-efficiency Digital Power Supply w/ PMBus 1.2
System Cooling: 4x 92x38mm 4-pin PWM middle fans, 2x 80x38mm 4-pin PWM rear exhaust fans
Front Panel:
Buttons: Power On/Off, System Reset
LEDs: Power, Hard Drive Activity, 2x Network Activity, System Overheat, Power Fail
Color: Black
Dimensions (WxDxH): 7.0 x 26.5 x 17.2 inch / 178.0 x 673.0 x 437.0 mm
Weight: 56.2 lbs / 25.5 kg
RoHS Compliant

[smicha]

Need a second coffee to get through this amazing stuff, Tutor. Fascinating!

[Tutor]

Need a second coffee to get through this amazing stuff, Tutor. Fascinating!

Smicha,

What do you think about my going the following route?
I'm considering soon purchasing the Supermicro X9DRX+-F, but not purchasing the Supermicro chassis, thus saving me over $900. I'm considering using some of my old stuff. I have two Lian LI PC-D8000 chassis [See http://www.lian-li.com/en/dt_portfolio/pc-d8000/ & http://www.overclockerstech.com/lian-li ... is-review/ ]. Each of my two Lian LI PC-D8000 chassis currently houses one of my two EVGA SR-2 systems. I'm considering selling both of my EVGA SR-2 motherboards that are each housed in those chassis. SR-2s still go for a hefty premium on Ebay, at least at (and more often that not over) their original 2010 -2011 purchase price. By selling the two SR-2s, I should hopefully recover all of the costs of the two Supermicro X9DRX+-Fs and the 22 PCI Express Riser Cards x8 to x16 Left Slot Adapter For 1U Servers which I will have purchased. Each chassis already has a Lepa G1600 PSUs and two 2T hard drives. I can install the 96 gigs of 2100+ MHz DDR3 ram in each of the SR-2s in each of the two Supermicro X9DRX+-Fs. I can use two of my four E5-4650 ES CPUs, still packaged, to operate each Supermicro X9DRX+-F, as if those CPUs were fast E5-2680 V1s. Thus, I would not have to expend immediately any more money for a chassis, ram, storage & associate cables, or PSUs and associated cables. Also, I have extra LGA 2011 low profile coolers from my updates to my four coolers on each of my two quad CPU Supermicro systems. I'm aware that I'll probably have to use my Dremel tools to set some new standoffs resulting from changing from an HPTX motherboard to a Supermicro motherboard with proprietary dimensions. I been up all night trying to get the basics of the upgrade at, eventually, no cost. What finally triggered this idea was my noticing last night for the first time that my Lian LI PC-D8000 chassis have 11 single wide PCIe slot connectors - the same number of single wide slots on the Supermicro X9DRX+-F. Also, I know that I haven't listed GPUs, but are there any other considerations that I've omitted? Does this plan make sense or am I too sleep deprived?

[smicha]

Question:

The Supermicro X9DRX+-F will be inside LianLi D8000, right?

What are these? Do you have any photos?
'the 22 PCI Express Riser Cards x8 to x16 Left Slot Adapter For 1U Servers which I will have purchased'

So do you plan to put 11 GPU into each of two D8000 using raisers (22 overall)? Will gpus be placed outside the case?