Best Practices For Building A Multiple GPU System

[Notiusweb]

Postby glimpse » Mon Feb 08, 2016 3:40 pm
Notius, could You provide test scene (or part of it) that is most likelly causing crash at Your system?

I have 1x splitter (backplate from Amfeltec), 4x GPU oriented & then risers 8X + motherboard runs on 16x, 8x & 1x native, also thunderbolt devices.. - I'd love to test things out on something that more than likelly to crash..
(feel free to drop PM if You would not like to share here).

Thanks Tom! I sent you a PM with the red car scene and settings I had making it crash. It will be interesting to see what happens when you replicate.

Post by Tutor » Mon Feb 08, 2016 8:06 pm
All excellent observations. I'll join in, but I will not be able to start until tomorrow, trying to replicate such usage on a MacPro, on a couple of Supermicros (a 32-core server [ X9QR7-TF+/X9QRi-F+ ] and a 16-core workstation [ X9DRX ]), and a Gigabyte or EVGA X79 system. It may take a few days to fit in tests on four of my systems.

You are right Tutor, Tom helping on Mac only peaked my interest in what might happen with Linux. Just kidding. I think 1 outside OS helps a lot.
Thanks guys!

[Tutor]

... . You are right Tutor, Tom helping on Mac only peaked my interest in what might happen with Linux. Just kidding. I think 1 outside OS helps a lot.
Thanks guys!

Notius, I need the test scene also.

[smicha]

Tutor,

Thank you so mych for D8000 image. My spying eyes tell me that if I put 360 at the back there is room for fans only at the top. I wish they thought about 2x360 at front instead of these massive hdd mounts.

[Tutor]

Tutor,

Thank you so mych for D8000 image. My spying eyes tell me that if I put 360 at the back there is room for fans only at the top. I wish they thought about 2x360 at front instead of these massive hdd mounts.

Smicha,

I forgot that you may have mentioned elsewhere using 360 fanW/radiators. Although it wouldn't surprise me if you've already done so, have you looked at these pics https://www.google.com/search?client=sa ... 8&oe=UTF-8 ? How many fans/coolers are you planning to connect? The Lian Li D8000 has 14 points to connect fans [<1 in. thickness] and thin [<1 in. thickness] radiators ("coolers or cooler equivalents"):
1) 4 cooler equivalents at the top [2 in. thickness total leeway];
4
2) 3 cooler equivalents at either side at front of case [2 in. thickness total leeway];
6
3) 1 cooler equivalents behind CPUs (right side - viewing case from rear - more depth flexible);
1 and
4) 3 cooler equivalents (left side - viewing case from rear- more depth flexible).

Assuming there are 7 GPUs, couldn't you use:
1) four cooler equivalents at top of case and
2) three cooler equivalents on right rear side of the case [viewing the case from the rear] or
if you are going to include the CPUs in the loop then use some side-front openings or
use externally a 9x cooler equivalent Watercool MO-RA3 Pro 9x120mm/4x180mm Extreme Radiator [ http://watercool.de/en/mo-ra3-0 ] or a combination thereof to cool it all ?

[smicha]

Tutor,

Thank you. I browsed all possible expansions of D8000. My plan was (not sure if we will change plans for 4 980ti on water and wait for pascal and avoid cutting 980ti and loosing warranty letting and the cards being sold easily in nearest future) to put a thick 360 radiator inside D8000 (top only for 4 fans for air intake, too few space for thicker rad) and external MORA3 420... Today we'll see what is out decision.

[Tutor]

Tutor,

Thank you. I browsed all possible expansions of D8000. My plan was (not sure if we will change plans for 4 980ti on water and wait for pascal and avoid cutting 980ti and loosing warranty letting and the cards being sold easily in nearest future) to put a thick 360 radiator inside D8000 (top only for 4 fans for air intake, too few space for thicker rad) and external MORA3 420... Today we'll see what is out decision.

Just found this, http://www.frozencpu.com/products/image ... ition.html , but haven't recently found any actually for sale.

[smicha]

Saw that too. But for 7 GPUs MORA is a must.

[smicha]

Tutor, my light in a tunnel What Supermicro board that has support for 4 Xeons fits in a Lian Li case?

[Tutor]

Tutor, my light in a tunnel What Supermicro board that has support for 4 Xeons fits in a Lian Li case?

Regrettably, none of them. Because 4 CPU motherboards, in the form factor that Supermicro makes them, have to support four air coolers and accordingly they (and their ram banks) have to be spaced far apart, making the motherboards for such systems large and often unusually proprietarily shaped, i.e., funky to me. Here are pics of my Supermicro 4 Xeon motherboard [ http://www.supermicro.com/products/moth ... R7-TF_.cfm ] [ 16.4" x 16.79" ], the case that it came in [ http://www.supermicro.com/products/syst ... R-TRF_.cfm ] = relatively high GPU IO space given only 4 PCIe slots; and where I sunk my duckets [ http://www.superbiiz.com/detail.php?name=SY-847R7FT ]. Here's the listing for the others - http://www.supermicro.com/products/moth ... 7000/#2011 .

By the way, the dimensions of the my X9DRX = 15.2" x 13.2."

P.S. The brains for the X9QR7-TF - http://www.cpu-world.com/sspec/QB/QBED.html & http://www.ebay.com/itm/Intel-Xeon-SAND ... 1641448357 . Those E5-4650 V1s have the same speed specs as the E5-2680 V1 and, in fact, in systems with fewer CPU slots appear in CPU-Z as E5-2680s v1s, even in my single CPU (made for i7) slot systems. Yes, I'm Mr. Cheapo - paid less for four QBEDs than I would have paid for one E5-4650 retail and the QBEDs have higher turbo boost max stage.

BTW - Since I first began conversing with you, you've been my light when I got deepest into the tunnel.

[smicha]

These two
http://www.superbiiz.com/detail.php?name=SY-847R7FT#
http://www.supermicro.com/products/syst ... R-TRF_.cfm

look so good. Thank you again.