Best Practices For Building A Multiple GPU System

Discuss anything you like on this forum.
Post Reply
User avatar
Tutor
Licensed Customer
Posts: 531
Joined: Tue Nov 20, 2012 2:57 pm
Location: Suburb of Birmingham, AL - Home of the Birmingham Civil Rights Institute

smicha wrote:Tutor,

IO on Asrock is also my concern. So if I remember well from your posts Supermicro i.e., X9DRX, X10DRX is not IO restricted/designed? So if Supermicro X10DRX is better way to go with I will do so - I may have access to X10DRX easily.
I cannot say for certain whether either the X9DRX or the X10DRX is absolutely IO unrestricted. Also, the specific GPU card(s) and the OS have to be taken into account. I can say that I know of no other motherboard(s) that support the same number of or a greater number of GPU cards as do the X9DRXs. Moreover, at the same time, the X9(10)DRXs act as standalone motherboards which is where my research indicates that the IO battle is waged on the motherboard. Maybe Trenton has figured out a way to stage some or all of the IO battle on it's chassis. I'd want Trenton to put me into contact with some excellent references before I spent close to $5,000 for a board that isn't fully a motherboard. That's the main reason why I recommended that you have Trenton put all of its cards on the table and stand behind, i.e., give you a refund upon failure of specific written claims/assertions of usefulness for a specific purpose before you spend thousands of dollars for that board. That way you can better assess your options and their likelihoods of success before you part with your money. Moreover, by attaching a string to your money before releasing it (i.e., getting a written the refund option), after you've parted with the money you stand a better chance of being able to draw the money back to you if the written claims aren't true.

Are you aware of anyone having successfully used the Trenton board in the same or similar fashion as you've laid out in your plans?
Because I have 180+ GPU processers in 16 tweaked/multiOS systems - Character limit prevents detailed stats.
User avatar
Tutor
Licensed Customer
Posts: 531
Joined: Tue Nov 20, 2012 2:57 pm
Location: Suburb of Birmingham, AL - Home of the Birmingham Civil Rights Institute

After giving the matter further consideration, can you connect the Trenton board to a ~$440 X9(or a more costly 10)DRX motherboard. That might provide better IO support for both reaching 20 GPUs as well as further expansion above 20 GPUs. I'll keep exploring whether their are better priced expansion boards that are similar to the Trenton board, since I'm in the process of migrating my old systems away from Amfeltec GPU Oriented Splitters and will seek better options for my future systems to maximize to the greatest extent possible data throughput.
Because I have 180+ GPU processers in 16 tweaked/multiOS systems - Character limit prevents detailed stats.
User avatar
smicha
Licensed Customer
Posts: 3151
Joined: Wed Sep 21, 2011 4:13 pm
Location: Warsaw, Poland

Tutor wrote:After giving the matter further consideration, can you connect the Trenton board to a ~$440 X9(or a more costly 10)DRX motherboard. That might provide better IO support for both reaching 20 GPUs as well as further expansion above 20 GPUs. I'll keep exploring whether their are better priced expansion boards that are similar to the Trenton board, since I'm in the process of migrating my old systems away from Amfeltec GPU Oriented Splitters and will seek better options for my future systems to maximize to the greatest extent possible data throughput.

That's great! Thank you. Hope EP2C612 can handle it too, but if DRX is the only (or best) option surely I'll go with it.
3090, Titan, Quadro, Xeon Scalable Supermicro, 768GB RAM; Sketchup Pro, Classical Architecture.
Custom alloy powder coated laser cut cases, Autodesk metal-sheet 3D modelling.
build-log http://render.otoy.com/forum/viewtopic.php?f=9&t=42540
User avatar
Notiusweb
Licensed Customer
Posts: 1285
Joined: Mon Nov 10, 2014 4:51 am

Hello, what do you use to track the power draw for a set of GPUs, and let's say you have 3 GPU, to 1 PSU, to an outlet distributor, you would attach the meter to the PSU before it goes to outlet distributor, right?

Just a side note, I added another Pascal X to my Titan X-only line up, which I use for other applications which more fully can exploit speed of Pascal.
Because my motherboard won't give me more than 13 GPU instances at a time, I'll run in one of 2 modes:
Mode 1 - motherboard Titan X and splitter-external 12 Titan Z (watercooled) - Mainly for Octane rendering
Mode 2 - motherboard Titan X, splitter-external 1 Regular Titan X, and 2 Pascal Titan X (all 3 air-cooled) - Mainly for other application scene creation, and some non-Octane rendering, and lower sampling Octane rendering.

What I have found is that the new Pascals, equally external aircooled with the 1 Regular Titan X, run a good 10-12 degrees cooler than the other 1 external Regular Titan X (ie 60F vs 72F) on a large Octane render single scene (1080p, 15000 s/px) project. I also can OC the Pascal +220 Mhz Core clock and it doesn't really seem to budge the temps more than 5 degrees (ie 65F). It would seem the newering of hardware potentially is going to have an added benefit of cooler cards. My Titan Z's when aircooled almost always ran hot 83 when OC'd, but then again they wuz old school.
Win 10 Pro 64, Xeon E5-2687W v2 (8x 3.40GHz), G.Skill 64 GB DDR3-2400, ASRock X79 Extreme 11
Mobo: 1 Titan RTX, 1 Titan Xp
External: 6 Titan X Pascal, 2 GTX Titan X
Plugs: Enterprise
User avatar
smicha
Licensed Customer
Posts: 3151
Joined: Wed Sep 21, 2011 4:13 pm
Location: Warsaw, Poland

Notiusweb wrote: ...What I have found is that the new Pascals, equally external aircooled with the 1 Regular Titan X, run a good 10-12 degrees cooler than the other 1 external Regular Titan X (ie 60F vs 72F) on a large Octane render single scene (1080p, 15000 s/px) project. I also can OC the Pascal +220 Mhz Core clock and it doesn't really seem to budge the temps more than 5 degrees (ie 65F). It would seem the newering of hardware potentially is going to have an added benefit of cooler cards. My Titan Z's when aircooled almost always ran hot 83 when OC'd, but then again they wuz old school.
Indeed. If I remember well 2xZ (28nm) drew 1350 W - entire system - (watercooled, OC to 1200mHz on core, score 440) but even on water were extremely hot - I could not touch backplates. 3x1070 (14nm) score around 420 but with power draw 350W. 1070 draws around 85W in Octane. 7x980ti (28 nm) draw 1350-1450W (watercooled), 7x1080 (14nm) - 900W. And yes - heat dissipation in 14nm Pascals is less, affecting ambient temp. less, cards are much cooler.
3090, Titan, Quadro, Xeon Scalable Supermicro, 768GB RAM; Sketchup Pro, Classical Architecture.
Custom alloy powder coated laser cut cases, Autodesk metal-sheet 3D modelling.
build-log http://render.otoy.com/forum/viewtopic.php?f=9&t=42540
User avatar
Tutor
Licensed Customer
Posts: 531
Joined: Tue Nov 20, 2012 2:57 pm
Location: Suburb of Birmingham, AL - Home of the Birmingham Civil Rights Institute

I use Kill-A-Watt EZ meters to keep track of electrical conditions. Where I plug the meter(s) in the electrical chain depends on what I'm trying to measure. For GPU watt measurement for rendering, I plug the meter in-line just after the computer's PSU's power cable with only a low power requirement video card, i.e., one of my GT 640s, installed and measure/review watts for that state. Next, I add the GPU(s) that I'm measuring that perform rendering and note any additional wattage required when the added GPUs are just running idle and next while they're rendering. I then compare that data. To measure total load to a circuit, I place the meter, as the last thing, into the power outlet and let all power consumers on that circuit run through the meter. Thus, in the example given by you, if all that's powering the GPUs is a PSU that's separate from the computer system's PSU, then I'd attach my meter just after the PSU's power cable.
Because I have 180+ GPU processers in 16 tweaked/multiOS systems - Character limit prevents detailed stats.
User avatar
smicha
Licensed Customer
Posts: 3151
Joined: Wed Sep 21, 2011 4:13 pm
Location: Warsaw, Poland

Tutor wrote:I use Kill-A-Watt EZ meters to keep track of electrical conditions. Where I plug the meter(s) in the electrical chain depends on what I'm trying to measure. For GPU watt measurement for rendering, I plug the meter in-line just after the computer's PSU's power cable with only a low power requirement video card, i.e., one of my GT 640s installed, and measure/review watts for that state. Next, I add the GPU(s) that I'm measuring that perform rendering and note any additional wattage required when the added GPUs are just running idle and next while they're rendering. I then compare that data. To measure total load to a circuit, I place the meter, as the last thing, into the power outlet and let all power consumers on that circuit run through the meter. Thus, in the example given by you, if all that's powering the GPUs is a PSU that's separate from the computer system's PSU, then I'd attach my meter just after the PSU's power cable.

This is exactly the same approach of mine.
3090, Titan, Quadro, Xeon Scalable Supermicro, 768GB RAM; Sketchup Pro, Classical Architecture.
Custom alloy powder coated laser cut cases, Autodesk metal-sheet 3D modelling.
build-log http://render.otoy.com/forum/viewtopic.php?f=9&t=42540
User avatar
Tutor
Licensed Customer
Posts: 531
Joined: Tue Nov 20, 2012 2:57 pm
Location: Suburb of Birmingham, AL - Home of the Birmingham Civil Rights Institute

Smicha, what up with Trenton board? I expect to have my air cooled 1070 build ready by earlier next week.

P.S. Trenton never returned any of my calls.
Because I have 180+ GPU processers in 16 tweaked/multiOS systems - Character limit prevents detailed stats.
User avatar
smicha
Licensed Customer
Posts: 3151
Joined: Wed Sep 21, 2011 4:13 pm
Location: Warsaw, Poland

Tutor wrote:Smicha, what up with Trenton board? I expect to have my air cooled 1070 build ready by earlier next week.
No response from Trenton so far. I am getting in touch with Asrock Rack to get components quotes only from
https://www.servethehome.com/asrock-rac ... er-review/
especially quotes for
Backplane Board : 3U8G_BPB1
Switch Board : 3U8G_SWB1
Interposer Board : 3U8G_IPB
http://www.asrockrack.com/general/produ ... C612#Plist
3090, Titan, Quadro, Xeon Scalable Supermicro, 768GB RAM; Sketchup Pro, Classical Architecture.
Custom alloy powder coated laser cut cases, Autodesk metal-sheet 3D modelling.
build-log http://render.otoy.com/forum/viewtopic.php?f=9&t=42540
User avatar
Tutor
Licensed Customer
Posts: 531
Joined: Tue Nov 20, 2012 2:57 pm
Location: Suburb of Birmingham, AL - Home of the Birmingham Civil Rights Institute

smicha wrote:
Tutor wrote:Smicha, what up with Trenton board? I expect to have my air cooled 1070 build ready by earlier next week.
No response from Trenton so far. I am getting in touch with Asrock Rack to get components quotes only from
https://www.servethehome.com/asrock-rac ... er-review/
especially quotes for
Backplane Board : 3U8G_BPB1
Switch Board : 3U8G_SWB1
Interposer Board : 3U8G_IPB
http://www.asrockrack.com/general/produ ... C612#Plist
The ASROCK RACK 3U8G-C612 8-WAY GPU SERVER reminds me a lot of my eight GPU Tyan server, except all of my GPUs in the old Tyan are in the rear.
Because I have 180+ GPU processers in 16 tweaked/multiOS systems - Character limit prevents detailed stats.
Post Reply

Return to “Off Topic Forum”