Best Practices For Building A Multiple GPU System

[Tutor]

Hello Smicha,

I'm sorry, but I forgot to answer a few of your other questions.

Tutor,

Tutor, does this mean you in USA have different circuit standards? 15A or 20A? Could you please give some hints about your PSUs you are using and how they are connected? BTW We must avoid voltage regulators http://www.220converters.com/Simran-Sym ... former.htm due to space restrictions and safety.

My house has quite a few 20A, but the vast majority are 15A.

My business has mostly 20A, but also quite a few 15A and quite a few 230 VACs. As to my PSUs (as well as GPUs), let's begin by my pointing out that if I were to get cut, I would bleed a Crimson Tide*/ of EVGA blood. Moreover, I have quite a few EVGA Tee Shirts with EVGA logos, many of which accompanied EVGA 1,600 W PSUs, most of which are now connected to 20A and 15A circuits.

*/ See, e.g., http://www.rolltide.com/index.aspx?path=football &
http://www.rolltide.com/watch/?Archive=8135

[smicha]

Hello Smicha,

I'm sorry, but I forgot to answer a few of your other questions.

Tutor,

Tutor, does this mean you in USA have different circuit standards? 15A or 20A? Could you please give some hints about your PSUs you are using and how they are connected? BTW We must avoid voltage regulators http://www.220converters.com/Simran-Sym ... former.htm due to space restrictions and safety.

My house has quite a few 20A, but the vast majority are 15A.

My business has mostly 20A, but also quite a few 15A and quite a few 230 VACs. As to my PSUs (as well as GPUs), let's begin by my pointing out that if I get cut, I bleed EVGA blood. Moreover, I have quite a few EVGA Tee Shirts with EVGA logos, many of which accompanied EVGA 1,600 W PSUs, most of which are now connected to 20A and 15A circuits.

EVGA PSUs are rebranded SuperFlower. When I opened a box with 1600W T2 it had exactly same cables as SF 1600W Titianium. EVGA-SF blood

[Tutor]

Further Suggestions To Increase A Motherboard’s Ability To Run More GPUs

First, the bios parameter with the greatest impact (previously referenced here many times) is "Above 4G Decoding." If it’s not activated, activate it now.
Also, under the Computer’s bios’ "PCIe/PCI/PnP Configuration" just as is "Above 4G Decoding" under that parameter (if it's revealed) may be other important GPU count extending parameters already exposed (or, if your motherboard’s bios does not expose these settings, then try to coax the manufacturer of your motherboard to provide a modified bios to expose these settings). Here are the other key settings to look for and to adjust:

1) MMIOH Base and
2) MMIO High Size.
They should be changed as follows:
MMIOH Base = 512G
MMIO High Size = 256G. */
Also, if there’s a bios boot feature for CSM mode **/ (or, if your motherboard’s bios does not expose this setting, then try to coax the manufacturer of your motherboard to provide a modified bios to expose this setting also), and then you should turn it "off." Then reboot. However, before turning CSM mode off, you may need to change in bios "onboard Video OPROM" to EFI and then reboot before you can CSM off. You may have to transition all available board UEFI feature selections to “on” to realize the maximum no. of GPUs that the motherboard can support.


*/ See - https://en.wikipedia.org/wiki/Memory-mapped_I/O ; http://stackoverflow.com/questions/5294 ... mapped-i-o ; and https://in.answers.yahoo.com/question/i ... 110AARNayR .

**/ See - https://neosmart.net/wiki/enable-uefi-boot/

[smicha]

Great tips Tutor - thank you!

BTW does Supermicro have MMIO options?

[Tutor]

Great tips Tutor - thank you!

BTW does Supermicro have MMIO options?

I'm told that some of their models do. The two models that I own don't. Those tips come from an acquaintance who has purchased at least eight of these NVIDIA GRID M40 – 4X MAXWELL GPUS/16GB RAM CARDS discussed here: https://www.servethehome.com/nvidia-gri ... ram-cards/ from here: http://www.ebay.com/itm/Nvidia-GRID-M40 ... SwqfNXoPLn . "Yes," that's four "light" Maxwell GPUs on each card for a total of 32 GPUs on eight cards. I'm still scratching my head and asking myself all sorts of questions such as "why did Nvidia come up with these card configurations?" , "What am I missing in not seeing their immediate desirability for what I/we do?", "Should I jump onboard immediately ?", "Can I get those GPUs working on a SuperMicro barebones system that I would want to purchase?" etc., etc., etc.

What are your thoughts about the Grid M40?

Update: There is no GRID M40 card. The 4 x GPU card is the early/maybe beta version of the Tesla M10, i.e., with 4G ram in the so-called Grid M40 vs. 8G ram in the Tesla M10 for each of the four GPUs on each card. "https://gridforums.nvidia.com/default/t ... mment=3264 - see in particular JasonSouthern's posts. "The NVIDIA Tesla M10 GPU accelerator works with NVIDIA GRIDTM software to provide the industry’s highest user density for virtualized desktops and applications. It supports 64 desktops per board and 128 desktops per server, giving your business the power to deliver great experiences to all of your employees at an affordable cost." http://images.nvidia.com/content/tesla/ ... nl-Web.pdf .

[smicha]

1600W P2 works like a charm in US on 7 GPUs - thank you guys for advice so much. And US citizens - don't do with 2000W PSUs

[Notiusweb]

Smicha -
You know what I like the most about your pic?
The cable ties!

LOL...Mine looks like such a mess, but it's really not worth it to tie up because it's not in a structured space like your build.
Knock on wood, but can I ask how do you leak test that? I would go nuts with that tight configuration!!!
1 false move and...Ugh...That's why I have such respect for your builds...! That attentiveness alone is worth a TON.
I in the past had to unwind everything for fear that there was water inside the card when I identified a leak on a run through of the coolant. I unwound everything ($#&* ! ! ! !), then I opened up the GPU from the waterblock and found nothing...In a way I was annoyed but actually I was relieved.

PS - Tutor, I asked ASrock to see if they could provide the extended PCI options you wrote about. Thanks for your post about the MMIO stuff!

[smicha]

Thanks!

I need to tighten and manage all cables for best possible airflow - it really matters. I've never had a leak in my life (except broken old shitty phobya radiator - EOL for good) probably due to soft tubing (I did only once hard tubing for 2Zs Piotr Adamczyk build and it really sucked - we changed it to ZMT and EK new pump mounts). So yes - I keep the machine at least a day or two for tests and not worried about leaks at all. And even if shit happens coolant is not conductive unless you add silver coil to it (I never do) - rads must be very well washed before first use - lemon, vinegar, shampoos, tap water, distilled water...

There are more important aspects - custom bios for 7 gpus, proper power, stability, fan speed adjustment,...

In this build I used for the first time farbwerk - when water is cold the rgb light gets blue, when hot - red - cool effect to show it off.

Coming next: attached

[smicha]

Guys,

Does anyone of you uses mac and pc on the same network so that they could share their drives each other? How to map mac os drives under windows?

[Tutor]

Guys,

Does anyone of you uses mac and pc on the same network so that they could share their drives each other?

Yes, I do.

How to map mac os drives under windows?

This [ http://www.digitaltrends.com/computing/ ... -mac-os-x/ ] best explains how I did do it; but there are other explanations : https://www.google.com/search?client=sa ... 8&oe=UTF-8 .