Best Practices For Building A Multiple GPU System

[Tutor]

Question:

The Supermicro X9DRX+-F will be inside LianLi D8000, right?

Right.

What are these? Do you have any photos?
'the 22 PCI Express Riser Cards x8 to x16 Left Slot Adapter For 1U Servers which I will have purchased'

See pic, below.

So do you plan to put 11 GPU into each of two D8000 using raisers (22 overall)?

My goal for my first build is to use those risers only to convert the slot size from x8 to x16; a) then use 6 (x16 to x16) cable risers (that recently came in) to be able to connect my six Titan Z GPU cards (away from the motherboard to spread them out, but still inside of the case); b) then use whatever combo of Amfeltec 4x GPU oriented splitters in two or more of the other 5 slots that would allow me to connect ten additional Titans ranging from my present 2xBlacks, my soon to be purchased 4xXs and leave space for4xPascals for next year. The second build would harness any overflow from my being too ambitious on the first one, as well as provide space for future GPU acquisitions (as will some of my other systems being emptied by these consolidations) as I coast deeper into my 60s. Other than Octane and the semi/current cap of 8 GPUs in Redshift [their work around until they are able to get it to handle more than 8 GPUs is that I have license to run additional instance of Redshift (using only one license) on the same system], TheaRender and Furryball Render have no caps per system.

Will gpus be placed outside the case?

Not my preference for this build. Some might be placed outside these 2 very deep chassis, but only if absolutely necessary. I plan to build a rack that will sit within the Lian Li to handle up to 18 GPU cards (whether single processor, dual processor or even triple GPU processor cards [probably being too overly optimistic here (and maybe elsewhere also) but I've always been that way].

[smicha]

If it works this will be the most powerful system out of a single motherboard. Good luck!

[Tutor]

If it works this will be the most powerful system out of a single motherboard. Good luck!

Thanks Smicha,
My guesstimate/rough projection for OctaneBench equivalency is that just with the addition of the four Titan Xs, I'd have a system capable of scoring over 2,400 points, since all of the GPUs will be Hydros or Hybrids. Four Pascal Hydros or Hybrids should take the system way over the 3,000 point mark, if Spring's hype is close to the truth. My goal for the end of 2016 is to have built 5 systems (1xTyan system, 2xSupermicro quadCPU systems and 2x Supermicro X9DRX+-F-O systems) that together have what is now an OctaneBench equivalency of over 100x GTX 980s, and would score, if it was permitted and possible, over 10,000 points together. But maybe I'll be able to realize that performance only with Redshift3d, Thea or Furryball renderers.

[smicha]

Have mercy, my friend

[Hydra]

The way you guys play with your hardware, its a wonder you don't go blind.

Have you considered modding a case so that it can make it easier to maintain?
http://www.engadget.com/2015/06/01/asus ... ansformer/

[Tutor]

The way you guys play with your hardware, its a wonder you don't go blind.

It's a short trip for me - I wear glasses.

Have you considered modding a case so that it can make it easier to maintain? ... .

Thanks for sharing. Hadn't previously thought of it, but no telling what this vid could lead me to do.

[Tutor]

Knowledge May Make Seeing Truer.
For, at least, a few of us we have to know how to make the GPU connections before we actually really see all of a motherboard's physical PCIe slot connectors. Otherwise, we might just quickly glance at them, immediately concluding that they are too small or blocked to be of any use to our mission. In a previous post of page 17 of this thread I said, “If it was a snake, it would have bitten me, given its close proximity.” Well, I now know that I was, at least, twice in range of being bitten. Originally, in the case of the PCIe slots in my systems, I fully saw only those slots that could be occupied directly by double width GPUs and weren't blocked by GPUs. This myopia caused me to view, for example, my LGA 1366 Gigabyte GA-EX58-UD5 and GA-X58A-UD3R systems as each only 3 GPU systems. Next, I became familiar with X16 to x16 risers, then all of a sudden I began to see that my Gigabyte GA-X58A-UD3R systems were potentially 4 GPU systems (one x16 PCIe slot was covered by a double wide GPU). Well, most recently I became familiar with x8 to x16 risers, x1 to x16 risers (pic shown below), x4 to x16 risers and corresponding extender riser cards such as, but not limited to, the PC1-X4 Extender Riser Card shown below and the x8 to x16 card pictured above. After becoming familiar with these connection devices, I went back and looked at the motherboards of my Gigabyte, as well as other, systems and discovered that I hadn’t mentally fully seen all of the PCIe connections because I hadn’t become aware of the physical means of making all of the possible connections. What I do know see is that my Gigabyte GA-EX58-UD5 systems have five PCIe slots (now seeing the additional x1 and the x4 slot per system) and my Gigabyte GA-X58A-UD3R systems have six PCie slots (now seeing the additional two x1 slots). So I mentally gained, although physically I aways had, 2 more GPU connection points on my Gigabyte GA-EX58-UD5 systems and 3 more GPU connection points (double what I had first seen) on my GA-X58A-UD3R systems. Whether all of these slots are actually suitable for making GPU compute card connections l’ll determine over the summer (and post updates). I may also try using Amfeltec splitters on one of each system and report comparative outcomes.

[Tutor]

My First Supermicro X9DRX+-F Build - Update

Tentatively, I've currently installed (1) one SATA host external card for connection to my two 4x external hard drive raid arrays, (2) two OCZ Storage Solutions RevoDrive 350 Series 960GB PCI Express Generation 2 x 8 Solid State Drives RVD350-FHPX28-960G (to run in raid 0), (3) two Amfeltec GPU Oriented x4 Splitter cards and (4) one 240G OWC Mercury Accelsior_E2 PCI Express SSD for running the system alternatively under another one of my favorite OSes when not needing to run Windows, still leaving me five completely empty PCIe slots (and remember that each of the two splitters allows me to connect 4 GPUs). What you don't yet see are the GPUs that I intend to install. I'm fabricating assemblies to store the GPUs internally inside of my Lian Li PC-D8000 chassis [ https://www.google.com/search?q=Lian+Li ... 60&bih=743 ]. Note that the Lian Li has a slide out motherboard tray [ http://www.lian-li.com/en/dt_portfolio/pc-d8000/ ] for ease of further customization and accommodates exactly eleven single-wide PCIe slots perfectly {I did have to add 5 motherboard supports, but otherwise the Supermicro X9DRX+-F motherboard fits perfectly}. Admittedly, I'm not sure at this stage whether I can leave all of those storage related cards connected and whether I can leave them connected and still have the system support 13 video cards (half of which will be Titan Zs) in an environment with 4 connectors (SplitterCard#1) + 4 connectors (SplitterCard#2) + 5 connectors (empty slots using x8 to x16 PCIe slot adapters which I hope will arrive this week from across the pond and then be connected to x16 to x16 powered riser cables). Only time and trial will determine that. Any run overs will go inside my second build.

[smicha]

Tutor,

Will you be using risers with this board? If so, what kind of risers?

[Tutor]

Tutor,

Will you be using risers with this board? If so, what kind of risers?

I intend to use traditional riser cables for the five open slots shown in my last pic, above. For those five empty slots, I'll be using x16 X x16 powered riser cables that look like these [ http://i.ebayimg.com/00/s/NjQzWDU5MA==/ ... L/$_14.JPG ]. Each of them will connect to the motherboard by via the x8 X x16 black and yellow riser cards shown in the pic above [ http://render.otoy.com/forum/download/f ... &mode=view ] that I posted just previously to the one above showing the motherboard and the Lian Li motherboard tray. The two Amfeltec GPU Oriented Splitters that I intend to use, and have already installed, act and will perform as risers of a sort because each of the four GPUs connected to each of them will not be connected directly to the motherboard. All of the GPUs must be raised to, at least, some degree because all of the GPUs have x16 size male connectors and all of the motherboard's slots are female x8 size.