Best Practices For Building A Multiple GPU System

Discuss anything you like on this forum.
Post Reply
User avatar
Tutor
Licensed Customer
Posts: 531
Joined: Tue Nov 20, 2012 2:57 pm
Location: Suburb of Birmingham, AL - Home of the Birmingham Civil Rights Institute

Notiusweb wrote:What about this, Titan-P?

http://hothardware.com/news/nvidia-gefo ... n-gtx-1080

Is this maybe the new Titan X Pascal?...

And here's the bigger question, with these less than expected specs:
WTF?
Regrettably, the article that you've referenced predates*/ the most recent announcement made by Nvidia of a Pascal GTX Titan X GPU. The "Full Fat" Pascal Titan referenced in the July 5th articles isn't the Titan X Pascal that will soon be released and it isn't the one that I want. I was hoping for a true GP100 Pascal GTX Titan. Now, that would be a "Full Fat" Pascal Titan - deserving of more than a now repetitive "X" designation. The specs for the officially announced Titan X Pascal aren't currently, for me, supportive of a $1,200 price tag. So, it appears that I'll continue sitting on the sidelines for the time being, making do with my assortment of pre-Maxwells, while awaiting a reasonably priced Titan with a full GP100, 3,840 cuda cores, and 12GB HBM2.


*/ It's dated July 05, 2016.
Because I have 180+ GPU processers in 16 tweaked/multiOS systems - Character limit prevents detailed stats.
User avatar
Notiusweb
Licensed Customer
Posts: 1285
Joined: Mon Nov 10, 2014 4:51 am

Tutor, any updates on your build, or same old same old? I have nothing new. Sometimes wondering if I changed my CPU could that allow me to go 14 GPU, but I never even get to BIOS with 14 , so I think probably no?
It's like that song from Sound of Music, "I am 16, going on 17,
Only for me it's, "I am 13, going on....damn 'd4 ' insufficient resources error again...that will lead us back to Doh-doh-doh-doh!..."
Win 10 Pro 64, Xeon E5-2687W v2 (8x 3.40GHz), G.Skill 64 GB DDR3-2400, ASRock X79 Extreme 11
Mobo: 1 Titan RTX, 1 Titan Xp
External: 6 Titan X Pascal, 2 GTX Titan X
Plugs: Enterprise
User avatar
Tutor
Licensed Customer
Posts: 531
Joined: Tue Nov 20, 2012 2:57 pm
Location: Suburb of Birmingham, AL - Home of the Birmingham Civil Rights Institute

Between real paying work, I've been working on (1) an upgrade to my MacPro4,1 which I had earlier updated to a MacPro5,1 and now I've been trying to get higher performance than a MacPro6,1 ("trashcan") and (2) an upgrade to my 6xTitan Z (primarily, but now has six other cards) system. I'm awaiting from across the pond more cables for the Z system so that I can finish reassembling it.
Because I have 180+ GPU processers in 16 tweaked/multiOS systems - Character limit prevents detailed stats.
User avatar
Notiusweb
Licensed Customer
Posts: 1285
Joined: Mon Nov 10, 2014 4:51 am

Yep, chugging along. Looking forward to Octane Bench 3.0. We'll be competing not only with 20 mixed GPU rigs, but some 8 or 9 all-Pascal rigs I'm sure... I think you, Tutor, will be the one to watch with 16-17
Then, henceforth...you shall be known as Darth Tutor. 8-)
Win 10 Pro 64, Xeon E5-2687W v2 (8x 3.40GHz), G.Skill 64 GB DDR3-2400, ASRock X79 Extreme 11
Mobo: 1 Titan RTX, 1 Titan Xp
External: 6 Titan X Pascal, 2 GTX Titan X
Plugs: Enterprise
User avatar
rappet
Licensed Customer
Posts: 1962
Joined: Fri Apr 06, 2012 3:57 pm
Location: The Netherlands
Contact:

Notiusweb wrote:Yep, chugging along. Looking forward to Octane Bench 3.0. We'll be competing not only with 20 mixed GPU rigs, but some 8 or 9 all-Pascal rigs I'm sure... I think you, Tutor, will be the one to watch with 16-17
Then, henceforth...you shall be known as Darth Tutor. 8-)
The curent 1260 OctanBench record score is already out of this planet ;)
Darth Tutor will break record and will be heading out of our solar system :lol:
Last edited by rappet on Sun Aug 21, 2016 11:24 am, edited 1 time in total.

4090+3089ti & Quad 1080ti
ArchiCAD25, ofcourse Octane & OR-ArchiCAD plugin (love it)
http://www.tapperworks.com
http://www.facebook.com/pages/TAPPERWOR ... 9851341126
http://www.youtube.com/user/Tapperworks/videos
User avatar
Tutor
Licensed Customer
Posts: 531
Joined: Tue Nov 20, 2012 2:57 pm
Location: Suburb of Birmingham, AL - Home of the Birmingham Civil Rights Institute

Sometimes, ideas have to marinade - viewtopic.php?f=40&t=43597&start=130#p228510 .
Notiusweb wrote:Yep, chugging along. Looking forward to Octane Bench 3.0. We'll be competing not only with 20 mixed GPU rigs, but some 8 or 9 all-Pascal rigs I'm sure... I think you, Tutor, will be the one to watch with 16-17
Then, henceforth...you shall be known as Darth Tutor. 8-)
rappet wrote: ... . The curent 1260 OctanBench record score is already out of this planet ;)
Darth Tutor will break his own record and will be heading out of our solar system :lol:
More importantly for me is to finally have consolidated what was 24 GPU centric systems, when I first started this thread, into many fewer systems ( I'm at 17 now ), i.e., under 15 such systems. I would like to have all of my present GPUs into just fourteen systems by this year's end ( if I'm blessed to reach age 63) and have room to add an additional ~ 20 GPU processors, such as upcoming Pascals and Volta's, and fit them into that 14 system count. Managing 24 systems was becoming my nightmare on Elm Street - those blade tipped fingers leave lasting, deep wounds. Admittedly, managing 17 GPU centric systems now is a lot easier than having to do it with 24 of them. For me, 15 or fewer such systems is the sweet spot, and my getting the maximum safe performance from them is just cake icing. What's next? My goal for age 65 (aren't I longevity optimistic) is to have consolidated all of my then possessed GPU processors into 12 such systems (at least 5 of which will be supporting a minimum of 20 GPU processors).

Here's that 2009 MacPro4,1 that I mentioned earlier. It's a work in progress. With the assistance of 2xAmfeltec GPU Oriented Splitters, I've loaded 10 cards into/onto a system with 1 double wide and three single wide PCIe slots = 2x x16/2x x4: Internally - 1xGTX 780-6G, 1x4 port e-SATA x8 card for externally connecting my San Disk 4xHD cases and 2xGPU-Splitters; Externally - 4xGTX 780-6G, 2x240G OWC 240GB Mercury Accelsior PCI Express Solid State Drive Cards in Raid 0, 1xSonnet Tango PCIe FireWire 400/USB 2.0 Adapter Card and 1xBlackMagic Design Intensity Pro Analog/HDMI Video Capture Card. All of the external cards, except for the GTX 780-6Gs, are x1 cards. I built an exhaust extension to take the heat away from the externally connected cards.

2nd thoughts: Might this pic be further evidence that Rappet is honing his mind reading {or, could it be spying} skills? This is the rocket/space ship that I'll be riding as I'm heading out of our solar system. P.S. Ignore my punching bag and vintage 2000 Toyota Celica GT-S in the background. I'm serious, "Don't look at them." Glad I didn't take the pic from the opposite side because then I'd be saying, "Ignore my thousand pounds of weights, weight bench and Teeter Hangup(side own)." And please - "No Batman or Bruce Wayne" references. I hang upside down (cape less) daily to feed my brain and to undo the back compression that heavy weightlifting causes. BTW: MacHeads are diehards and Klingons die hard! Being Darth would place me in the wrong movie and/or time/space quadrant.
You do not have the required permissions to view the files attached to this post.
Because I have 180+ GPU processers in 16 tweaked/multiOS systems - Character limit prevents detailed stats.
User avatar
Notiusweb
Licensed Customer
Posts: 1285
Joined: Mon Nov 10, 2014 4:51 am

Tutor can try his 12 GPU rig right now to try and top my 1260, but I think he's waiting to decimate it on Obench3.
On Obench3 I'll have 13 going, as I'll have my Regular Titan X as the 13th. All things same as V2, I would venture my score ~1360.

Did you know on Daz Studio, IRay renderer, and separately through other apps, on Indigo Renderer Beta 4, you can use all GPU as well, PLUS your CPU (the only limit is your imagination...). However both are slower than Octane, which means you have to have a good imagination too...

Tutor, is that where the rig resides, or just the building point? I didn't know a Mac could house so many GPU :P , but if there's anyone I know that would MACify a rig to its fullest, it would be a member of the High Council :ugeek:
Qapla'!
Win 10 Pro 64, Xeon E5-2687W v2 (8x 3.40GHz), G.Skill 64 GB DDR3-2400, ASRock X79 Extreme 11
Mobo: 1 Titan RTX, 1 Titan Xp
External: 6 Titan X Pascal, 2 GTX Titan X
Plugs: Enterprise
User avatar
Tutor
Licensed Customer
Posts: 531
Joined: Tue Nov 20, 2012 2:57 pm
Location: Suburb of Birmingham, AL - Home of the Birmingham Civil Rights Institute

First, do not overlook incentivizing your imagination. Then, throw your rock into the pond that interests you and marvel at the ripple effect. Don't be surprised if some fellow/subsequent traveler retrieves that rock and discovers it to be made of gold. Don't have any regret that you didn't notice the rock's ultimate value, but rejoice in the finder's having a keener eye than did you, for that rock that you once held is now brightening someone else's journey. Moreover, you did help to create a wave!
Notiusweb wrote:Tutor can try his 12 GPU rig right now to try and top my 1260, but I think he's waiting to decimate it on Obench3.
On Obench3 I'll have 13 going, as I'll have my Regular Titan X as the 13th. All things same as V2, I would venture my score ~1360.
I tweak my GPUs [and CPUs] to compete only against one individual and that individual and I share the same shell. If someone likes my tweaking's effects, I'll gladly tell any of them how I did it, getting into the minutest of detail. My ultimate goal is to top my last best score. That's not to say that your Titan Z scores didn't play a part in my understanding of the cards' potential for tweaking. It did. The scores that I list in my signature or elsewhere are to inspire others of like mind to compete against their past scores by showing them what an old man can do as a baseline. There are trade-offs in competing against only one's self, but I'll take those trade-offs gladly considering that I like managing my own self-improvements.
Notiusweb wrote:Did you know on Daz Studio, IRay renderer, and separately through other apps, on Indigo Renderer Beta 4, you can use all GPU as well, PLUS your CPU (the only limit is your imagination...). However both are slower than Octane, which means you have to have a good imagination too...
Thanks for this information. Keep feeding us [ https://www.youtube.com/watch?v=L7SkrYF8lCU ]. As I've posted in this thread earlier, there are other 3d rendering applications that also do not artificially impose GPU limits; but I didn't know about the leeway of the applications that you list. Technological/actual use limitations are to be expected. I think it's admirable when the maker of a 3d application comes right out and says "Our testing shows that X app can handle at best x no. of the GPUs that we've tested [and list them specifically], but you're welcome to try all of your GPUs that you can muster." I just don't want my provider to secretly limit my ability to realize all that X app has to offer me because the provider is secretly [or semi-secretly or even openly] in competition with me - the customer.

As far as imagination goes, one just needs to treat the brain as a sacred vessel: remember to constantly feed the brain physically (such as by good diet and exercise) and use it constantly (such as engaging in constant learning , becoming maximally curious about everything imaginable and savoring the excitement in using the brain to tackle the myriad of tasks we face). Having/marshalling proper emotions and goals also play a part in having a good imagination. For example, when one devotes his/her time to helping others achieve their dreams, imagination is given proper purpose and becomes inspired by the greater goods which are everlasting, outliving our individual shells because the good we do becomes intergenerational. Chose to not allow bad emotions, such as (but surely not limited to), pettiness, selfishness, defeatism, etc. to affect our relations to others, i.e., our immediate family and, yes, our worldwide family consisting of all humanity. For example, my posts related to this build has the potential to benefit many: (1) those who want to sell old MacPro systems, (2) those who want to own a MacPro for 3d work, but cannot afford a MacPro 2013, but can afford an older (most likely used) MacPro, (3) those who sell 3d rendering software, such as, e.g., Otoy, (4) those who sell GPUs, (5) those who wish to sell old Intel processors, etc. Some might say that's just a tiny impact, but every little bit helps even if it helps just one person, especial if that person pays it forward by helping someone else and the chain of being of assistance to others continues onward. That the kind of synergy can help to further spur our creativity/imagination once we realize how powerful our imagination can be in impacting the goals, hopes, desires and lives of others. A mind, without imagination revealed and the fruits of imagination passed on, is a terrible thing to waste. Notiusweb, it's obvious from looking at your Titan Z system's highest and other Octane Bench scores that you haven't been wasting your mind, that you have a very productive imagination, that you have have excellent mental faculties and that you often compete against yourself. Wasn't that 1260 Octane Bench score not your first try in bench marking your GTX Titan Zs? Wasn't there no one other than yourself against whom you could have been in competition? I'm not waiting to decimate your score with even more GPUs. That wouldn't even be a fair comparison - 12 apples vs. 12 apples plus an an orange, a banana and a some grapes isn't a fair comparison. Your score could be higher if you further chilled your system's coolant as I have suggested to you earlier, assuming that you get that ailing Titan Z GPU treated. I will post some more of my scores in the future, but certainly with the hope and expectation that someone else will come along to best all of those scores on an apple to apples basis and further educate me about how I can further improve my systems' performances. It's the give and take that leads to our long term growth and through our humility and aid to and from others that we will prevail for the long term.
Notiusweb wrote:Tutor, is that where the rig resides, or just the building point? I didn't know a Mac could house so many GPU :P , but if there's anyone I know that would MACify a rig to its fullest, it would be a member of the High Council :ugeek:
Qapla'!
The pre-trashcan MacPros were built to house just one double-wide GPU as evidenced by there being only 2 six-pin PCIe GPU powering points and only one double wide slot. Going beyond Apple's intent by deploying a second double-wide GPU internally would not only require adding another power source for the second GPU, but would also when using a 1xSix-pin and 1xEight-pin GPU card occupy/cover both the second (a single-wide) x16 PCIe slot and the adjacent single-wide x4 PCIe slot, leaving only the last single-wide x4 PCIe slot empty. A couple of years ago on MacRumors' MacPro site I set forth a method of loading internally a third double-wide PCIe GPU, but to be done properly that involved sacrificing/removing three of the hard drive bays and modifying the drive support bracket, as well as adding additional power resources. I like the method that I've employed this time best. No hard drive bays are sacrificed, the GPUs are allowed to be spaced further apart and aren't heated by system internals or exhausted heat, the Apple intended PCIe compartment stays cooler, the extra GPUs can run at the maximal intended speed and I can run more GPUs and not have to sacrifice using other cards that assist in 3d and video production processes. On the downside, I lose one of the two DVD drive bays because I have, for now, installed in it's place a 450W BoosterX power supply to power the internal GTX 780-6G; however, I might, in the end, use a beefier external power supply to power the internal GPU if I decide to stick with only four external GTX 780-6Gs and save space for my second DVD drive. If, on the other hand, I decide to go with six GTX 780-6Gs (i.e., five external and one internal) for this build, then I'll also likely stick with the BoosterX PSU in one of the DVD drive bays. Of course in the near future, I'd like to substitute those 780-6Gs with five or six GTX 1080 TIs, and then use those GTX 780's to update one of my three older MacPro2,1s (circa- early 2007), after getting them to run the latest MacOS supportive of the 780-6Gs. I also intend to pass down GPUs now in other newish systems to the other MacPros as I add newer GPUs to those newish SandyBridge systems. I've installed the top of the line CPUs for their era in the 2007 MacPros and I have installed de-lidded X5690s CPUs in the MacPro4,1 (2009), making the 2009 system about as fast for most CPU related chores as would be the 2013 trashcan MacPro. Isn't GPU rendering technology great for those who can profitably put it to use on old hardware to preserve system longevity and investment costs?

The MacPro4,1's present location is just a building point. I use portions of my garage/basement not only for storing vehicles, but also for doing some others of my chores, such as building systems, car repairs, home repair project staging, exercising my body, preparing for outdoor planting, etc. So, in essence, this area consists of my laboratories. That MacPro system when completed will go into my studio and the next system I chose to upgrade will go onto one of the workbenches for improvements. System improvement has been, still is and will likely be an ongoing process that I'll have to fit in-between render jobs.

BTW - All of us are members of our High Council and I cherish every one of our meetings.
Because I have 180+ GPU processers in 16 tweaked/multiOS systems - Character limit prevents detailed stats.
User avatar
Notiusweb
Licensed Customer
Posts: 1285
Joined: Mon Nov 10, 2014 4:51 am

There is a new series of higher scores putting 10 GTX 980s (9 Ti's and 1 regular 980) closing in on my 12 Zs...
The 980/980 Ti, while 2nd in ultimate power compared to Titan X, has to be the best value card.
Win 10 Pro 64, Xeon E5-2687W v2 (8x 3.40GHz), G.Skill 64 GB DDR3-2400, ASRock X79 Extreme 11
Mobo: 1 Titan RTX, 1 Titan Xp
External: 6 Titan X Pascal, 2 GTX Titan X
Plugs: Enterprise
User avatar
Tutor
Licensed Customer
Posts: 531
Joined: Tue Nov 20, 2012 2:57 pm
Location: Suburb of Birmingham, AL - Home of the Birmingham Civil Rights Institute

The sum is dictated by each part. Key: To maximize the sum, first separately maximize the potential of each individual part - this might involve tweaking each GPU individually using, for example a software utility such as MSI Afterburner or EVG Precision X */, or in extreme cases for the extremists only (like me and especially in the case of this MacPro build where there's no MSI, EVGA or other comparable software utilities) require fiddling with otherwise hard(ware) setting [You're own your own and you (and only you) assume all risk - https://www.youtube.com/watch?v=i6vjBwtSvCQ and http://www.overclock.net/t/1474548/kepl ... t-it-means */]; then make only universal settings that allow each part to show it's fullest potential when combined with the whole.
Notiusweb wrote:There is a new series of higher scores putting 10 GTX 980s (9 Ti's and 1 regular 980) closing in on my 12 Zs...
The 980/980 Ti, while 2nd in ultimate power compared to Titan X, has to be the best value card.

My guess is that those GPUs are air cooled and over-clocked little to none, inadequately powered and/or if over-clocked and adequately powered, then they're probably throttling. The basis for my guess is that I can could run 10 of my 780 Tis air cooled (to max) and overclocked, adequately powered, yielding a score of about 1,184 for 10 of them and about 1,421 for 12 of them (average per single GPU = ~118.4; btw - I've had that score posted in my signature for many months and its been posted here - ) and after I converted them to hybrids and using the same settings generally (e.g., GPU fan speed irrelevant), yielding a score of about 1,230 for 10 of them and about 1,476 for 12 of them (slightly better than the score for my six Titan Zs) (average per GPU = ~123). Furthermore, the highest posted score for a single over-clocked GTX 980 TI is 141.99; so 10 of those, if adequately cooled and throttling was overcame, as well as adequately powered when running 10 of them, should score about 1,419.9 and 12 of them should score about 1,703.88. I haven't keep up on the current price of the Titan Z relative to that of the 980 TI, so I don't currently have a basis to say which is the best value. But, after having built a number of systems, I can say that I tend to prefer dual GPU processor cards especially when they're water-cooled because of system space, slot no. limitation, system cooling, system power, and individual GPU cooling and power requirements. All of these become a major considerations when dealing with twice the no. of video cards. Thus, I prefer building multiple GPU systems using GPUs with two (and someday hopefully more) processors on the video card. Maybe someday we can just get the GTX cards with only the compute power functionality for reduced pricing, with video output functionality being an additional cost offering.

*/] See - https://www.youtube.com/watch?v=SKWbKCKsWVQ .

P.S.
1: I've bought a few video cards where the video output capability was and is still broken (terms of purchase were - try compute performance and accuracy before payment). Boy! were they cheap; but so am I. All of those displayless video cards still compute like champs. As should be obvious, the reason why the card doesn't display video matters much.
2: My scores referenced above were achieved without any hardware bios modifications; but such mods are upcoming, especially for the MacPro rebuilds first, but the mod for the Mac itself has to be done on one of my Windows systems.
Last edited by Tutor on Thu Aug 25, 2016 12:43 am, edited 1 time in total.
Because I have 180+ GPU processers in 16 tweaked/multiOS systems - Character limit prevents detailed stats.
Post Reply

Return to “Off Topic Forum”