Best Practices For Building A Multiple GPU System

[smicha]

[Tutor]

Here's a guide that I found recently for overclocking multiple GPUs (even those that aren't connected to a display) under Linux: http://unix.stackexchange.com/questions ... tions-cuda .

[glimpse]

Here's a guide that I found recently for overclocking multiple GPUs (even those that aren't connected to a display) under Linux: http://unix.stackexchange.com/questions ... tions-cuda .

Thanks for info Tutor! As always something to absorb =) But I have somewhat related question..

I know You're good with Macs. Could You recomend the best way to OC cards on Mac? Or let's say a way/tool to flash 780s so they would run faster without additional overclock? Like out of the box?

[Tutor]

Here's a guide that I found recently for overclocking multiple GPUs (even those that aren't connected to a display) under Linux: http://unix.stackexchange.com/questions ... tions-cuda .

Thanks for info Tutor! As always something to absorb =) But I have somewhat related question..

I know You're good with Macs. Could You recomend the best way to OC cards on Mac? Or let's say a way/tool to flash 780s so they would run faster without additional overclock? Like out of the box?

For Mac (or other) tweakers with older GTX GPUs [ http://pastebin.com/ZR5RHRxU ] - visit https://www.cindori.org/software/ and check out Freya and Zeus.

For Mac (or other) tweakers with more recent GTX GPUs, follow these guides - http://forum.netkas.org/index.php/topic ... l#msg16969 and/or http://www.tomshardware.com/faq/id-1865 ... cards.html and use http://www.softpedia.com/get/Tweak/Vide ... itor.shtml .

For those who want someone else to do the tweaking, contact MacVideoCards - See, e.g., http://www.macvidcards.com/store/c2/Nvidia_GPUs.html .

[glimpse]

Thanks Tutor! I'll take look into those for deeper inspection. Could have google out smh myself, but I'm not that much into macs & it's hard to distil what is good & what is useless content over there. Thanks for those links, Man.

[Notiusweb]

Hi Tutor, do you think Linux will have the ability to use/adapt somehow the advancing GPU functions of DirectX 12 and NVLink.
Also, how goes it with the rig I think you are also doing some water cooling right?
-Regards!

[Tutor]

Hi Tutor, do you think Linux will have the ability to use/adapt somehow the advancing GPU functions of DirectX 12 and NVLink.

Hello Notiusweb,

Your continuing efforts to maximize the performance of your system, is most inspiring. I thank you and all others who have helped me to increase my knowledge regarding how my consolidations [see, below] should be made to best satisfy my rendering needs.

From what I’ve read, DirectX12 will allow Linux, MacOS and Windows (as well as the tablet and phone OSes) and associated applications to benefit from DirectX12’s allowance of more direct control of GPU hardware features. Driver support is essential. Nvidia’s driver development appears to be fully on board towards this goal [ http://blogs.nvidia.com/blog/2014/03/20/directx-12/ ], so I have no reason to doubt that Linux application programmers will be able to take advantage of DirectX12’s benefits. However, I’m not certain about application support timelines and intended scope of coverage.

NVlink in another animal that's been very elusive. I'm not certain what hardware it requires for full implementation, i.e., whether it will be scaled down to take advantage of PCIe hardware or will require newer and much more expensive hardware [ http://blogs.nvidia.com/blog/2014/11/14/what-is-nvlink/ ]. From what I've read about NVLink, I'm doubtful that PCIe users will benefit from this technology; and this is particularly so since full implementation of this new technology also appears to be tied to IBM CPUs.

Also, how goes it with the rig I think you are also doing some water cooling right?
-Regards!

Regarding the consolidation of the bulk of my systems’ GPU processors, my plan thus far has been to consolidate the vast majority of my GPUs from 24 systems to twelve systems: 4 MacPros and 9 self-builds. I have been able to devote time to planning and purchasing most of the parts for the consolidations, but as I have come to realize that the actual, full implementation of the consolidations may necessitate additional purchases. Nine of my systems (excluding the four MacPros {and excluding four CPU exclusive Windows and Linux [self-built] rendering systems}) will have no fewer that six CUDA GPU processors. Those nine systems will have an average of 12 -13 GPU processors each, ranging from 6 to 16 GPUs. Only 2 will initially have less than 10 GPUs. I intend to add additional GPUs when Pascals drop and later - Voltas. So that may require my rearranging some of them to maximize performance. My goal is to initially water cool at least five of those systems. However, the crush of 3d rendering projects has caused me to delay completing my consolidations up to now. Such 3d rendering projects have historically tended to decrease enough (to be meaningful) beginning in mid-Nov. & continuing to the end of Dec. of each year, only to bounce back beginning the new year. So I’ve put off completing the majority of my consolidation efforts until the lull occurs. I’ll keep all of you updated regarding the outcomes and any new lessons learned.

[Tutor]

Here're links to other posts about IO memory mapping and the desirability of above 4G memory mapping for large GPU installations (I'll add more to this post as I find more good resources):

1) http://hardforum.com/showthread.php?t=1860102 and see especially (also cited in hardforum thread) https://en.wikipedia.org/wiki/PCI_hole ;
2) https://communities.intel.com/message/292070 ;
3) viewtopic.php?f=25&t=45279&start=70#p240481 ;

[Notiusweb]

Tutor, on the topic of > 12 GPU, where does the Thea and FurryBall limit sit (or is there a limit)? Also, thoughts on speed (vs Octane if it serves to simplify a description)?
BTW, just thoughts, did you ever see Indigo Render, it is nice but only allows 1 GPU as CUDA at the present time. NVidia Iray, at least for Daz Studio, allows 13 GPU, but is slow. Bunkspeed Zoom looks like it is good at doing cars, but who knows about the rest (animations, characters, effects, etc...)

[glimpse]

Bunkspeed Zoom looks like it is good at doing cars, but who knows about the rest (animations, characters, effects, etc...)

bunkspeed is based on the same iRAY- simplified interface it far less flexible..I've tried it before Octane & did not like it at all..