Best Practices For Building A Multiple GPU System

Discuss anything you like on this forum.
Post Reply
User avatar
Seekerfinder
Licensed Customer
Posts: 1600
Joined: Tue Jan 04, 2011 11:34 am

Hey Tutor,
Most of our contributions here pale in comparison to what you add in these forums... It would be great if Smicha could transfer from one rabbit hole to the other without coming to the surface at all. But any shortcut strategy involves more risk of course.

Keen to hear the results. I think that once it works though the words 'do not touch' should be laser cut into each side panel...

Seeker
Win 8(64) | P9X79-E WS | i7-3930K | 32GB | GTX Titan & GTX 780Ti | SketchUP | Revit | Beta tester for Revit & Sketchup plugins for Octane
User avatar
smicha
Licensed Customer
Posts: 3151
Joined: Wed Sep 21, 2011 4:13 pm
Location: Warsaw, Poland

Guys,

You are all awesome. I really appreciate your willingness to help. If I had the machine it would take me no more than 2 hours to drain the loop, remove them all, reset bios, at test them one by one and do all the stuff, including asus flashback which I've done dozens of times. Believe me -removing the water of the loop takes 5 minutes, removing the 7-way bridge takes no more than 10-15 minutes. So (for me) it's pretty quick and easy. The problem is that I have to assist the guy remotely and first test all other possibilities.

Tomorrow morning we're testing putting back power cables. If this is not good we'll go with the drainage. I'll report you on all.
3090, Titan, Quadro, Xeon Scalable Supermicro, 768GB RAM; Sketchup Pro, Classical Architecture.
Custom alloy powder coated laser cut cases, Autodesk metal-sheet 3D modelling.
build-log http://render.otoy.com/forum/viewtopic.php?f=9&t=42540
User avatar
Seekerfinder
Licensed Customer
Posts: 1600
Joined: Tue Jan 04, 2011 11:34 am

smicha wrote:The problem is that I have to assist the guy remotely and first test all other possibilities.
Oh... I missed this part. Bummer. It's obviously not great for the end user but I hope your client at least appreciates that that system is an unusual power house and that some snags are likely.
Seeker
Win 8(64) | P9X79-E WS | i7-3930K | 32GB | GTX Titan & GTX 780Ti | SketchUP | Revit | Beta tester for Revit & Sketchup plugins for Octane
User avatar
Tutor
Licensed Customer
Posts: 531
Joined: Tue Nov 20, 2012 2:57 pm
Location: Suburb of Birmingham, AL - Home of the Birmingham Civil Rights Institute

Does wood conduct electricity? Yes, but it depends on moisture content [ https://www.quora.com/Does-wood-conduct-electricity ]. Teflon is even less conductive. So a teflon coated stand might be better than a wooden crate to raise a bunch of GPUs - just make sure the teflon coating covers all areas that might come into contact with GPUs and check the coating's rated temperature resistance and compare that to the environmental temperatures of the other heat contributors in the area of concern. You don't want any teflon or other insulator melting to occur.

conductivity of copper: 59,600,000 S/m
conductivity of graphite: 300 to 50,000 S/m
conductivity of wet wood: 0.001
conductivity of dry wood: 0.000000000000001 S/m
Conductivity of teflon: 0.000000000000000000000001 S/m

See also, http://www.lehigh.edu/~amb4/wbi/kwardlo ... tivity.htm .
Because I have 180+ GPU processers in 16 tweaked/multiOS systems - Character limit prevents detailed stats.
User avatar
smicha
Licensed Customer
Posts: 3151
Joined: Wed Sep 21, 2011 4:13 pm
Location: Warsaw, Poland

I am preparing something more scientific - if you guys could help with some data it would be great
viewtopic.php?f=11&t=53959
3090, Titan, Quadro, Xeon Scalable Supermicro, 768GB RAM; Sketchup Pro, Classical Architecture.
Custom alloy powder coated laser cut cases, Autodesk metal-sheet 3D modelling.
build-log http://render.otoy.com/forum/viewtopic.php?f=9&t=42540
User avatar
Notiusweb
Licensed Customer
Posts: 1285
Joined: Mon Nov 10, 2014 4:51 am

Anyone imagine what 8x Pascal GTX 1080 would score on Bench? :?:
Win 10 Pro 64, Xeon E5-2687W v2 (8x 3.40GHz), G.Skill 64 GB DDR3-2400, ASRock X79 Extreme 11
Mobo: 1 Titan RTX, 1 Titan Xp
External: 6 Titan X Pascal, 2 GTX Titan X
Plugs: Enterprise
User avatar
glimpse
Licensed Customer
Posts: 3740
Joined: Wed Jan 26, 2011 2:17 pm
Contact:

Notiusweb wrote:Anyone imagine what 8x Pascal GTX 1080 would score on Bench? :?:
if OctaneBench numbers (after optimisations) will corelate to SP performance we are looking at 1200 out of the box & close to 1500 watercooled/Oced =) hitting 1000 would happen with 5-6 cards.
User avatar
smicha
Licensed Customer
Posts: 3151
Joined: Wed Sep 21, 2011 4:13 pm
Location: Warsaw, Poland

3090, Titan, Quadro, Xeon Scalable Supermicro, 768GB RAM; Sketchup Pro, Classical Architecture.
Custom alloy powder coated laser cut cases, Autodesk metal-sheet 3D modelling.
build-log http://render.otoy.com/forum/viewtopic.php?f=9&t=42540
User avatar
Notiusweb
Licensed Customer
Posts: 1285
Joined: Mon Nov 10, 2014 4:51 am

glimpse wrote:
Notiusweb wrote:Anyone imagine what 8x Pascal GTX 1080 would score on Bench? :?:
if OctaneBench numbers (after optimisations) will corelate to SP performance we are looking at 1200 out of the box & close to 1500 watercooled/Oced =) hitting 1000 would happen with 5-6 cards.
I'm gonna say Titan X : GTX 1080 = 1:1.25 (my ball park is based off of some articles comparing FPS)
So, if 8 X scored 1,096 on bench, then 1,096 * 1.25 = 1,370.

So, I'm going to say for GTX 1080, 8 cards will hit 1,370.

Note that this is V2. It will be interesting to see if V3 actually scores higher/lower with same scenes in a bench run...
Win 10 Pro 64, Xeon E5-2687W v2 (8x 3.40GHz), G.Skill 64 GB DDR3-2400, ASRock X79 Extreme 11
Mobo: 1 Titan RTX, 1 Titan Xp
External: 6 Titan X Pascal, 2 GTX Titan X
Plugs: Enterprise
User avatar
Notiusweb
Licensed Customer
Posts: 1285
Joined: Mon Nov 10, 2014 4:51 am

Tutor, what say you? (assume PSU and power is not a concern or limitation, and assume I have most optimal BIOS for adding GPU):

I am trying to max cards on rig from 13 to 14, but under my current arrangement I always get a d4 Out of PCI Resource error. Fine, okay...but in this arrangement my Titan X is connected to slot 1 at 16x.
Could halving my 16x connection of my Titan X to 8x alone allow me to add another card?
Or, would IO space say, "No, irrespective of how many lanes are used, I can only delegate so much address memory to so many devices. It's a device count thing, not a lane usage thing...
If you have 13 at 1x, 13 at 4x, 13 at 8x, or 13 at 16x...I don't give a F&$%...you're only getting 13...because that's all I can allocate address memory to...so F&$% off with the 14, you damn F&$%ing piece of S&*$..."
Win 10 Pro 64, Xeon E5-2687W v2 (8x 3.40GHz), G.Skill 64 GB DDR3-2400, ASRock X79 Extreme 11
Mobo: 1 Titan RTX, 1 Titan Xp
External: 6 Titan X Pascal, 2 GTX Titan X
Plugs: Enterprise
Post Reply

Return to “Off Topic Forum”