Best Practices For Building A Multiple GPU System

[smicha]

This baby supports up to 80 pice lanes
http://www.trentonsystems.com/single-bo ... host-board

[Tutor]

Hi Tutor.

I've seen the link You've dropped there a while ago =) & I've been curious how that works..the biggest question I have..why does Windows in some cases see cards, but we can't use them? Cos that might be the case - machine OS is goign to recognise those & we still would not be able to use them..

Curious to see You future experiments on this topic, & seems You have the same mobo, what is promissing to say the least =)

Smicha,

that rack fil of Phis is stuning! as for backplates, there are some products, like those from Trenton 18 x16=DDD

Hello Glimpse,

I don't have the same mobo, but I plan to get a few of them ASAP. Now the hard part is waiting for those x8 to x16 converters or cables to swim across the ponds.

[Tutor]

This baby supports up to 80 pice lanes
http://www.trentonsystems.com/single-bo ... host-board

Smicha,

Still reviewing the Trenton option/data, but at first glance, the info seem much dated (8 G ram max - got to be kidding with 12 G GPUs).

[smicha]

This baby supports up to 80 pice lanes
http://www.trentonsystems.com/single-bo ... host-board

Smicha,

Still reviewing the Trenton option/data, but at first glance, the info seem much dated (8 G ram max - got to be kidding with 12 G GPUs).

The board’s eight standard DIMM sockets can support a total system memory capacity of 512GB when using the very expensive and relatively rare 64GB DDR4 memory DIMMs, 256GB when using the moderately expensive 32GB DDR4 memory DIMMs, and 128GB for the more common, and less costly 16GB DIMMs.

[Tutor]

This baby supports up to 80 pice lanes
http://www.trentonsystems.com/single-bo ... host-board

Smicha,

Still reviewing the Trenton option/data, but at first glance, the info seem much dated (8 G ram max - got to be kidding with 12 G GPUs).

The board’s eight standard DIMM sockets can support a total system memory capacity of 512GB when using the very expensive and relatively rare 64GB DDR4 memory DIMMs, 256GB when using the moderately expensive 32GB DDR4 memory DIMMs, and 128GB for the more common, and less costly 16GB DIMMs.

Thanks, I got side tracked by following some of their internal links that loads 2008 info. Up-to-date system info looks very promising,. Do you have any pricing?

[smicha]

From https://bitcointalk.org/index.php?topic=64450.0

Trenton BPX6806 backplane + $600.00
Trenton NLT6313 host board + $210.00


So I assume that overall cost (mobo + backplane) shall not exceed $1k.

[Tutor]

http://www.supermicro.com/products/chas ... -R1620.cfm w/dual 1620W Redundant Power Supplies for about $1K (USD). Still most of 12 GPUs would have to be installed externally. Best combo, which I consider to be lacking when it comes to my planned installation, is the $1,552 (USD) - Supermicro SuperServer SYS-7047R-TXRF Dual LGA2011 1280W 4U Rackmount/Tower Server Barebone System (Black) [ http://www.supermicro.com/products/syst ... r-txrf.cfm ] that comes w/dual 1280W Redundant Power Supplies. It's a barebones system - so no CPUs, storage or ram. Although the motherboard itself has on-board video, it's not useful to CUDA users when it comes to running CUDA based apps like Octane. To stall and to run Nvidia drivers, an Nvidia GPU has to be the display device.

[Tutor]

This baby supports up to 80 pice lanes
http://www.trentonsystems.com/single-bo ... host-board

That's because it's a dual LGA 2011 Xeon system ( as is Tommes' ) [ "The memory controller and 40 PCI Express (PCIe) lanes are integrated on the CPU. " - http://en.wikipedia.org/wiki/LGA_2011 ] . The more PCIe slots you have filled with GPUs, the fewer the theoretical number of lanes available to be allocated to each GPU. I used the term "theoretical" because other devices are going to use some of those lanes also.

Once upon a time, I used to be a well-known Xeon clock-tweaker ( inventing a method of giving my systems 13 -14 bins of turboboost potential by under-clocking the system's CPUs at idle and minimal load and making the CPUs turboboost hellaciously under full load), but my days of massive Xeon tweaking ended on Intel's LGA 2011 Xeon and later Xeon systems. With the predecessors Xeon CPUs, like LGA 1366 Nehalems and Westmeres, if I clock-tweaked my system's CPUs, the only other feature that got automatically increased (and it was by the same percentage) was my system's memory speed - so I either used much faster rated memory or told a lie to bios by indicating that my system's memory was a lot slower than it actually was - so that when truboboost mode kicked in, my memory would boost no higher than it's rated speed or if I wanted it to boost higher than standard LGA 2011 memory speeds (1333 MHz), I had to use faster rated memory - I usually used 2000 MHz for lots of headroom. With LGA 2011 Xeons and later Xeons, Intel tied the same clock that used to control only CPU and memory speed, to the PCIe bus, thus tying it also to not only video, but also to devices like USB, SATA, etc. (i.e., all the other types of devices that are PCIe bound [and also use PCIe lanes]). Now the maximum theoretical safe over-clock is less than 20% of what is used to be - and that is now 1.0755% of the CPUs rated clock speed. Only certain Supermicro computers (the DAX family, in particular, that Supermicro released for Day Traders and other high speed freaks (me included)) can attain that new theoretical ( but current really tiny) maximum.

[Tutor]

Hi Tutor.

... .why does Windows in some cases see cards, but we can't use them? Cos that might be the case - machine OS is goign to recognise those & we still would not be able to use them... .

If Windows sees all of the cards in a system or otherwise connected to the system's PCIe bus and the Devise Manager doesn't report that the card has been disabled because it wasn't functioning properly, then there are potentially only certain other potential causes: 1) Octane has a lesser GPU limit (I do not yet have any hard evidence to believe that Otoy has hobbled Octane to see less that 12 GPUs per system), 2) the particular Octane build being used has a flaw ( I've yet to encounter this problem ) ( I'd run Octane using various Octane builds to test this possibility) , 3) the Nvidia drivers (video and/or CUDA) didn't install properly for each GPU ( I've had this to happen) (I'd serially install each GPU and the video and CUDA drivers for each card each time also performing the registry hack describe within the discussion of problem no. 9 in the Octane User Manual), or 4) the particular build of the Nvidia driver(s) has a flaw ( I've had this to happen) ( I'd try another build - just as I did in the past).

[Tutor]

... .
I also found such a monster
http://www.asrockrack.com/general/produ ... C612#Plist
... .

That monster costs (over $4K (USD)) which is more than the cost new of the Tyan that I recently sold. Both are designed for 8 double wide GPUs, except the AsRock is designed for high end GPUs and my Tyan that I recently sold was designed for GTX cards specifically. My other nickname is Mr. Cheapo.

I do, however, like the Trentons and since their facility is just a couple of hours from where I reside (Georgia and Alabama are bordering states), I'll be calling them on Monday to get more specifics.