Best Practices For Building A Multiple GPU System

Discuss anything you like on this forum.
Post Reply
User avatar
Tutor
Licensed Customer
Posts: 531
Joined: Tue Nov 20, 2012 2:57 pm
Location: Suburb of Birmingham, AL - Home of the Birmingham Civil Rights Institute

glimpse wrote:Tutor, check this out! =)
500x1000px-LL-afb265f8_DSC00001.jpeg
remember I've told You 'bout aqua freezers? Well the Guy use one with Aquacomputer passive rad =) do a reverse (image search) if You're interested =)..don't remember where from I got this image, but as I found it accidentally on my PC, though of sharing this with You =)

Glimpse,

Hope you're having a wonderful time. Thanks for the lead. It took me to here: http://www.aquatuning.us/water-cooling/ ... =1&sSort=4 . However, mainly because of your pep talks, I still haven't given up on getting Mr. Freeze(r) to do what I want it to do. Again, Thanks to you (and to everyone else who has given me input and inspiration on this quest). What a wonderful family I have! It's a true blessing.
Because I have 180+ GPU processers in 16 tweaked/multiOS systems - Character limit prevents detailed stats.
User avatar
glimpse
Licensed Customer
Posts: 3740
Joined: Wed Jan 26, 2011 2:17 pm
Contact:

Tutor wrote:
Glimpse,

Hope you're having a wonderful time. Thanks for the lead. It took me to here: http://www.aquatuning.us/water-cooling/ ... =1&sSort=4 . However, mainly because of your pep talks, I still haven't given up on getting Mr. Freeze(r) to do what I want it to do. Again, Thanks to you (and to everyone else who has given me input and inspiration on this quest). What a wonderful family I have! It's a true blessing.

hehe, I had this video for a while saved out somewhere in my bookmarks =)
https://www.youtube.com/watch?v=Du25KvkpaBM (not the best quality, but..)
I believe You've seen it as You posted the link in previous go =)
User avatar
Tutor
Licensed Customer
Posts: 531
Joined: Tue Nov 20, 2012 2:57 pm
Location: Suburb of Birmingham, AL - Home of the Birmingham Civil Rights Institute

glimpse wrote:
Tutor wrote:
Glimpse,

Hope you're having a wonderful time. Thanks for the lead. It took me to here: http://www.aquatuning.us/water-cooling/ ... =1&sSort=4 . However, mainly because of your pep talks, I still haven't given up on getting Mr. Freeze(r) to do what I want it to do. Again, Thanks to you (and to everyone else who has given me input and inspiration on this quest). What a wonderful family I have! It's a true blessing.

hehe, I had this video for a while saved out somewhere in my bookmarks =)
https://www.youtube.com/watch?v=Du25KvkpaBM (not the best quality, but..)
I believe You've seen it as You posted the link in previous go =)
What the owner of that system describes as the effect of that chiller in the video, sounds somewhat familiar, doesn't it? Someone who I sleep with posted, " I believe that I will be able to boost the freezer's cooling capacity when the GPUs are under load to keep them at about 55 degrees Fahrenheit (12.78 degrees Celsius) and otherwise maintain them at about 40 degrees Fahrenheit (4.44 degrees Celsius) when the GPU's aren't under load." [ http://render.otoy.com/forum/viewtopic. ... 7&start=30 ]. He's still believing what he believed earlier. Again, thanks for these kicks in the rump so that I might then be able to post here [ http://render.otoy.com/forum/viewtopic. ... 45#p224445 ] with supporting pics, that each of two of my water cooled systems achieves as follows:

Temp = low teens
Pathtracing
Pathterm=1
Coherent Ratio=1

~400 Ms/s

Temp = low teens
Directlighting
Pathterm=1
Coherent Ratio=1

~800 Ms/s

I can dream big, can't I!
Because I have 180+ GPU processers in 16 tweaked/multiOS systems - Character limit prevents detailed stats.
User avatar
smicha
Licensed Customer
Posts: 3151
Joined: Wed Sep 21, 2011 4:13 pm
Location: Warsaw, Poland

I am impressed with the chiller and the temperature results. As a 'silence searcher' I see it perfect if only I could place the chiller in another room. Tutor, are you saying you'll go with this solution?
3090, Titan, Quadro, Xeon Scalable Supermicro, 768GB RAM; Sketchup Pro, Classical Architecture.
Custom alloy powder coated laser cut cases, Autodesk metal-sheet 3D modelling.
build-log http://render.otoy.com/forum/viewtopic.php?f=9&t=42540
User avatar
glimpse
Licensed Customer
Posts: 3740
Joined: Wed Jan 26, 2011 2:17 pm
Contact:

smicha wrote:I am impressed with the chiller and the temperature results. As a 'silence searcher' I see it perfect if only I could place the chiller in another room.
this stuff is usually used by aqua fanatics who keep sea animals (some expensive fish, corals etc =) - actually buying these units from aqua shops might be a bit cheaper too =)

Again, the sound it makes..- is far from pleasant =)
User avatar
Tutor
Licensed Customer
Posts: 531
Joined: Tue Nov 20, 2012 2:57 pm
Location: Suburb of Birmingham, AL - Home of the Birmingham Civil Rights Institute

smicha wrote:I am impressed with the chiller and the temperature results. As a 'silence searcher' I see it perfect if only I could place the chiller in another room. Tutor, are you saying you'll go with this solution?
Smitcha,
What I should have said more clearly is that I haven't given up on my getting my mini-freezer to replicate on my H2O loop the success of the chiller that Glimpse referenced. Right now, that particular chiller in Glimpse's pic is just to expensive in the model of it that I'd need to cool 14 GPU processors.


Glimpse,
On a different, but somewhat related, note, the two Tyan 8xGPU Servers that I have are the old Intel 1366 dual CPU models. Since they have been EOLed, there will not be any further bios developments for them. I'm running the last bios that those systems will have. The person who I recent spoke with at Tyan indicated that those systems, in fact, do have an 8 GPU processor limit and pointed out that those systems were meant to use then Tesla cards, most of which didn't even have display capability and dual processors when these systems were released. GTX video cards usually use more IO space than Tesla cards and even getting 8 GTX cards to work properly might not work and, of course and as I'd already read that, the successor Tyan 2011 dual CPU/8 GPU chassis has a version that's specifically designed to run 8 GTX cards and there's another version for Tesla cards. In sum, running 8 of my 780 Ti ACX SC OCs in my Tyan causes 4 of my 18 ram sticks to be disabled. That reminded me of Amfeltecs caution to users of their external chassis -
"To be able support 7-8 GPUs on the general purpose motherboard sometimes requested disable extra peripherals to free up more IO space for GPUs." Apparently, the GTX 780 TI ACX SC OC is a IO space hog by comparison to the other GTXs, because I can run 8 original Titans, 4 Titan Zs, 8 GTX 780 6G, etc. in my Tyans without experiencing any memory/IO space issues. Thus, at most, I can run 8 GPU cards each with a single processor in one of my Tyans, or 4 dual processor GTX GPU cards in either of them, or other permutations/combinations that add up to 8 or fewer GPU processors. That's it. However, with 4xGTX Titan Zs there, I end up with 4 empty slots. Trying to operate my Tyans by filling those 4 empty slots with other PCIe cards such as for SATA, SSDs etc. isn't even an option because the system will not boot with them installed because of IO space limits. To make a long story a bit shorter, I'm contemplating just installing the eight GTX cards (that together have 14 GPUs and that I'd originally planned to be in one system, i.e., 6XGTXTitan Z's, 1xGTX Titans, and 1xGTX Titan Black) into two other systems (and only put single processor GPU cards in both of my Tyans) so that no PCIe slot space goes wasted in either of them. I plan to use my water cooling system to cool both of those systems' GPUs. In other words - with the GPUs I'd originally planned be installed and water-cooled in one Tyan system, I'll split them up into 2 groups and water-cool them in 2 Gigabyte-UP4 systems.

P.S. Glimpse, I also realized that my water flow rate was to high using the Tyan because of its fan controller settings. Thus, Mr. Freeze(r) didn't have the opportunity to cool the water sufficiently. By moving the GPUs to my Gigabyte-UP4 systems, I expect that I'll be better able to adequately lower the flow rate and, thus, the cooling of the water.
Because I have 180+ GPU processers in 16 tweaked/multiOS systems - Character limit prevents detailed stats.
User avatar
FrankPooleFloating
Licensed Customer
Posts: 1669
Joined: Thu Nov 29, 2012 3:48 pm

NewEgg currently has Rosewill HERCULES-1600W PSUs on sale for $269 ($130 off). Not a bad a deal...
Win10Pro || GA-X99-SOC-Champion || i7 5820k w/ H60 || 32GB DDR4 || 3x EVGA RTX 2070 Super Hybrid || EVGA Supernova G2 1300W || Tt Core X9 || LightWave Plug (v4 for old gigs) || Blender E-Cycles
User avatar
Tutor
Licensed Customer
Posts: 531
Joined: Tue Nov 20, 2012 2:57 pm
Location: Suburb of Birmingham, AL - Home of the Birmingham Civil Rights Institute

FrankPooleFloating wrote:NewEgg currently has Rosewill HERCULES-1600W PSUs on sale for $269 ($130 off). Not a bad a deal...
Frank,
Thanks for update.
Because I have 180+ GPU processers in 16 tweaked/multiOS systems - Character limit prevents detailed stats.
User avatar
Tutor
Licensed Customer
Posts: 531
Joined: Tue Nov 20, 2012 2:57 pm
Location: Suburb of Birmingham, AL - Home of the Birmingham Civil Rights Institute

The Amfeltec Cluster transport is x1; but Amfeltecs GPU splitters [~$200 (USD) - http://amfeltec.com/products/flexibl...-gpu-oriented/ - are x4; so how about this:

Just create an L shaped GPU support rail that has a "L" lip with holes for screws to attach up to 4 GPU cards ; attach it to the motherboard side of the system and attach the GPUs to the back on the L-lip; attach the GPUs to the splitter cards [put the splitter card in the x8 or 16 slot] and now you've got 4-x4 GPUs externally spaced wide apart, in the open air, blowing hot air upward and externally feed, being blown air at their base from the fan cooling the PCIe bay; and three internal PCIe slots for non-GPUs (MPs support only 4 GPU processors - they're four slotted). GPUs would be much cooler too. It's now easier to connect them to a powerful external PSU. Metal railing would be dirt cheap; PSU and GPUs would cost what GPUs and PSU costs. All welcomed to pick up and run with this? Also the bracket could be placed lower given the design of Windows motherboards and case PCIe placement.. All welcomed to pick up and run with this?

P.S. - If you have a Windows system that has adequate IO space to support up to 7 or 8 GPUs, just make the bracket longer with enough connection points for however many GPUs your system will support and get 2 splitter kits; of course that will use two internal PCIe slots.
You do not have the required permissions to view the files attached to this post.
Because I have 180+ GPU processers in 16 tweaked/multiOS systems - Character limit prevents detailed stats.
User avatar
glimpse
Licensed Customer
Posts: 3740
Joined: Wed Jan 26, 2011 2:17 pm
Contact:

Tutor wrote:The Amfeltec Cluster transport is x1; but Amfeltecs GPU splitters [~$200 (USD) - http://amfeltec.com/products/flexibl...-gpu-oriented/ - are x4;
noticed this "nonsense" I mean their GPU Cluster can hook up to 16 cards (in four stand configuration with proper host card), but why on earth they made that only x1???

That splitter is way more interesting, but again - those wires could be slightly longer..Overall nice Idea =)

I've been thinking to hook splitter on x4 slot & place cards on the backside of the case, mounted like fins (no 90 degree to the side of case, but like 30..)
Post Reply

Return to “Off Topic Forum”