Best Practices For Building A Multiple GPU System

[Tutor]

By the way, the Supermicro X9DRX+-F has low profile CPU slots. So, one'll need passive (a cooler block)/active (i.e., fan/H20 coolers) that fit Narrow+ILM+Processors. Here's two examples: [ http://www.dynatron-corp.com/en/ipsearc ... Processors ]. The ones that I'm using look and perform more like version R14. They were previously used in my quadCPU/32 core Supermicros until I recently upgraded the coolers in them. Underneath each cooler is an E5-4650 v1 ES CPU (8-core). These ES versions (purchased on Ebay for about $500 ea.) were the last build before the final released version. Mine have an extra bin for turbo boosting, i.e, they turbo boost up to 3.6 GHz from 2.7 GHz base. They are recognized by non-quad CPU systems as E5-2680 v1s, except that my chips turbo one bin higher than the E5-2680 v1s [ http://www.cpu-world.com/CPUs/Xeon/Inte ... -2680.html ]. That's why I always use active cooling with them.

P.S. The E5-2680 v2 have that extra turbo bin to 3.6 GHz [ http://www.cpu-world.com/CPUs/Xeon/Inte ... %20v2.html ] and thus can be used safely with passive coolers. The E5-2680 v3s are 12 cores that turbo to 3.3 GHz [ http://www.cpu-world.com/CPUs/Xeon/Inte ... %20v3.html ]. But the v3 Xeons aren't compatible with this motherboard.

[smicha]

Tutor,

Will you be using risers with this board? If so, what kind of risers?

I intend to use traditional riser cables for the five open slots shown in my last pic, above. For those five empty slots, I'll be using x16 X x16 powered riser cables that look like these [ http://i.ebayimg.com/00/s/NjQzWDU5MA==/ ... L/$_14.JPG ]. Each of them will connect to the motherboard by via the x8 X x16 black and yellow riser cards shown in the pic above [ http://render.otoy.com/forum/download/f ... &mode=view ] that I posted just previously to the one above showing the motherboard and the Lian Li motherboard tray. The two Amfeltec GPU Oriented Splitters that I intend to use, and have already installed, act and will perform as risers of a sort because each of the four GPUs connected to each of them will not be connected directly to the motherboard. All of the GPUs must be raised to, at least, some degree because all of the GPUs have x16 size male connectors and all of the motherboard's slots are female x8 size.

Thank you for such precise answer. So, if I understood you well, to connect e.g. 10 titans X to the motherboards I shall use exactly same risers?

[Tutor]

Tutor,

Will you be using risers with this board? If so, what kind of risers?

I intend to use traditional riser cables for the five open slots shown in my last pic, above. For those five empty slots, I'll be using x16 X x16 powered riser cables that look like these [ http://i.ebayimg.com/00/s/NjQzWDU5MA==/ ... L/$_14.JPG ]. Each of them will connect to the motherboard by via the x8 X x16 black and yellow riser cards shown in the pic above [ http://render.otoy.com/forum/download/f ... &mode=view ] that I posted just previously to the one above showing the motherboard and the Lian Li motherboard tray. The two Amfeltec GPU Oriented Splitters that I intend to use, and have already installed, act and will perform as risers of a sort because each of the four GPUs connected to each of them will not be connected directly to the motherboard. All of the GPUs must be raised to, at least, some degree because all of the GPUs have x16 size male connectors and all of the motherboard's slots are female x8 size.

Thank you for such precise answer. So, if I understood you well, to connect e.g. 10 titans X to the motherboards I shall use exactly same risers?

Yes, for that motherboard you could use ten x16 X x16 powered riser cables connected to ten x8 X x16 black and yellow riser cards connected to 10 PCIe slots, leaving you with one empty PCIe slot; or you could use two Amfeltec GPU Oriented x4 Splitters in two of the eleven slots (connecting up to four video cards to each of the two Splitter cards and use two x16 X x16 powered riser cables connected to two x8 X x16 black and yellow riser cards connected to two more PCIe slots, leaving seven empty slots (and this is what I'd do), or another combination thereof.

[Tutor]

NOTE THAT I'VE ADDED TO THIS POST ON JULY 8, 2016

Four excellent resources for learning to tackle IO space issues are (1) your system's block diagram (usually depicted early on in your motherboard's user manual), showing the layout of the system's CPU(s), PCIe slots, DMI points, other features/resources/peripherals and their connections (see example pics below) [ At the very least, if you try to follow Amfeltec's caution that " ... on the general purpose motherboard sometimes requested disable extra peripherals to free up more IO space for GPUs," then {whether or not you're actually using any Amfeltec product} you may have a better idea of which peripherals to disable];
and (2) these three articles which will take you in-depth into the factors that you need to consider when purchasing a system for GPU rendering and to get the most GPUs fully operational in your system:
(a) http://resources.infosecinstitute.com/p ... nsion-rom/
(b) http://resources.infosecinstitute.com/s ... d-systems/ and
(c) http://resources.infosecinstitute.com/s ... d-systems/ .

For example, from the pic of the Supermicro X9DRX+-F's motherboard layout I would tend to first try connecting more of my dual GPU video cards (such as Titan Zs) to PCIe slots 5, 6, 7, 9, and 10 that are directly connected to CPU2, than in PCIe slots 1, 2, 3, 4, and 8 that are directly connected to CPU1 because CPU1 has been allocated a lot more DMI chores than CPU2 (which - from the factory - has none).

[Notiusweb]

Tutor,
You mentioned using Amfeltec splitters. Would you say that, in terms of IO space from the motherboard, would you possibly be better suited to use a x16 lane for a splitter vs a x8 lane. Does this only affect speed and not maximum use of IO space?

[Tutor]

Tutor,
You mentioned using Amfeltec splitters. Would you say that, in terms of IO space from the motherboard, would you possibly be better suited to use a x16 lane for a splitter vs a x8 lane. Does this only affect speed and not maximum use of IO space?

Ten of the eleven PCIe slots are x8 electrically and in size and the eleventh slot is x8 in size and x4 electrically, so I don't have that option. But, I fully understand your concern about speed. For 3d rendering the speed of the slot is much, much less of a noticeable factor than is the speed of the GPU. Slot speed determines only how long it takes to feed render data to the GPUs and those are such small time increments (i.e., a few seconds) that they won't be noticeable to me. The x8 PCIe slots in the new system are V3. x8 V3 is as fast as x16 V2. Over eighty percent of the slots's that I've been using in my other systems are not V3 and more of those non-V3 slots are x8 speed than are x16 speed. In fact, three of my four MacPros are the 2007 version and they're V1 PCIe, which is half as fast as V2, which is half as fast as V3 and I've (so far) preferred doing most of my designing on one of my old MacPros and haven't really noticed any significant difference in total render times amounting to more than 4-5 seconds for rendering large format frames in a few cases and more often only a couple of seconds. However, one of the reasons that I chose to purchase the Amfeltec splitters over the Amfeltec chassis is that the chassis operates at x1 speed and the splitters operate at speeds four times faster (i.e., x4 speed). The most important reason was, however, the lower costs of the splitters. Maximum IO space potential isn't affected at all by slot speed.

[Notiusweb]

Here is another one, and I apologize if already discussed in this post. This is something that bitcoin forums are always discussing - power. And not PSU power supply, but actual power to the outlet.

For your rig, which will at minimum be above average in power usage, are you planning on using a 15 amp breaker outlet, or 20?
Will you be using any software, i.e. MSI Afterburner, to cap off some of the power usage?
Extension chords to other breakers to distribute up the power?
Water cooling?

I noticed heavy scenes (higher res, more meshes, multiple lighting instances, hair) are really good at raising temps really quickly, and even with the best possible PSUs, multi-GPU systems like to test circuit breakers.

[Tutor]

Here is another one, and I apologize if already discussed in this post. This is something that bitcoin forums are always discussing - power. And not PSU power supply, but actual power to the outlet.

For your rig, which will at minimum be above average in power usage, are you planning on using a 15 amp breaker outlet, or 20?
Will you be using any software, i.e. MSI Afterburner, to cap off some of the power usage?
Extension chords to other breakers to distribute up the power?
Water cooling?

I noticed heavy scenes (higher res, more meshes, multiple lighting instances, hair) are really good at raising temps really quickly, and even with the best possible PSUs, multi-GPU systems like to test circuit breakers.

Apologize? Oh contraire! Thank you for the mind meld - two or more heads can be much better than one. My most important purpose in contributing to (and starting) this thread was/is to bare all of my thoughts regarding multiple GPU systems for all of similar interest to savor, affirm, correct, question, critic, build upon, expand, limit, etc. to provoke the readers to lay their minds bare to help improve all of our outcomes. So, PLEASE ask any question(s) that you have even if it/they might have been already touched upon in this thread because nuance matters to our having a complete understanding of relevant issues. Moreover, reasoning and reason(s) matter. Our questions may require each of us to think more deeply about issue(s) from different, but vital, angles. The very nature of our endeavor to force a system to use ever possible resource to handle every GPU demands that we force our minds to become more resourceful by fully considering all issues from all relevant angles. Because our rigs require above average power usage, all issues regarding powering many GPUs such as, but no limited to, electrical safety and GPU performance must be fully considered.

For this particular rig, I’ll be using 1x 1600W LEPA PSU and 3x 1,600W EVGA PSUs, each connected to a separate/different surge protected power strip with each power strip connected to a separate/different 15 Amp circuit breaker. Only components related to this system will be connected to each such breakers. I’ll be using wall socket connected wattage meters to measure power draw under worst case scenarios for all powered system components, including monitors, water-cooling, etc., distributing discrete components that have user selectable power connections so that the draw under worst case scenarios remain at a peak draw at or under 1750W or 14.6 Amps total per breaker. I’ll be using 12 gauge or lower gauge extensions cords and power strips rated for 15 Amp, 125 V, 1875 maximum watts, with the shortest lengths possible for safe travel and connection. I’ll be conducting periodic/weekly breaker testing, subject to project completion requirements. I intend to use MSI Afterburner to tune my GPUs for maximum safe performance while staying within my self-imposed 1750W peak power total usage cap. I intend to water-cool all of the GPUs in this build. I will maintain the room temperature at or under 78 degrees Fahrenheit. Is there anything that I’ve missed or not covered? What are your recommendations or what would you do differently? If you have any other questions, just ask them.

[Tutor]

For those interested in learning more about IO space, please note my additions to this post: http://render.otoy.com/forum/viewtopic. ... 71#p241271 .

[Tutor]

I’m considering purchasing three One Stop Shop CA 160024U Compute Accelerators [see http://www.onestopsystems.com/4u-comput ... tx-titan-x and http://www.onestopsystems.com/sites/def ... 364-4u.pdf ]. Each Accelerator can be filled with 16 GTX Titans Xs {or even better 16 GTX Titan Pascals} (4 GPUs in each of four canisters in each Accelerator), for a total of 48 GTX Titan Xs (or Pascals), for four Supermicro servers. Each Accelerator connects to one [or up to four] host server(s) via four PCIe x16 Gen3 cables. Also, each Accelerator has three 3000W PSUs, but since the third PSU is for redundancy, two such PSUs provide only about 6,000W. My plan is to add 12 GPUs to each server. As to my eleven slot servers systems, that will leave me with enough empty slots in each of them for four OCZ Storage Solutions RevoDrive 350 Series 960GB PCI Express Generation 2 x 8 Solid State Drives that can each yield read speeds upwards of 1800MB/s, writes of 1,700MB/s and 4K random write IOPS of 140,000 and up to 6,800MB/s read and write speeds in raid 0.