Best Practices For Building A Multiple GPU System

Discuss anything you like on this forum.
Post Reply
User avatar
Seekerfinder
Licensed Customer
Posts: 1600
Joined: Tue Jan 04, 2011 11:34 am

smicha wrote:Tutor,
Huge thank you for this great thread and you massive help. The score is based on Linux and these GPUs are not overclocked .... I know too little about Linux for now to OC GPUs. Core clocks on water are 1900mhz and with slight OC to 2050 I think 2300 score seems to be achievable. I am on Ubuntu 14.04 lts.is there any software as MSI afterburner to easily OC GPUs in Linux?
I'll post on the forum details from the build.

A short video for now.

https://youtu.be/Uk2qIYvgFaE
Smicha
Awesome job, Smicha!

Which Xeons did you go for? We're looking at E5 2687W V4's for our next build. But that's because we also want the machine to act as a workstation (3ghz+).

Seeker
Win 8(64) | P9X79-E WS | i7-3930K | 32GB | GTX Titan & GTX 780Ti | SketchUP | Revit | Beta tester for Revit & Sketchup plugins for Octane
User avatar
Tutor
Licensed Customer
Posts: 531
Joined: Tue Nov 20, 2012 2:57 pm
Location: Suburb of Birmingham, AL - Home of the Birmingham Civil Rights Institute

To Smicha and all the rest of my friends who've bitten the Linux bullet -

Here's the Linux GPU Overclocking course that I recommend that you take (if clock tweaking floats your boat):

(1) Watch Osirez Tech's two videos starting here [ https://m.youtube.com/watch?v=jVm0epOZDa0 ] on youtube. Interestingly, you'll find that the overclocking ability is given to you by Nvidia as a part of the Linux driver download, but that you have to code its operation yourself [ then read this - https://www.phoronix.com/scan.php?page= ... 0.28-Linux - and this - http://us.download.nvidia.com/XFree86/L ... tions.html - Particularly this part of the latter reference: Option "Coolbits" "integer" ]. Osirez illustrates this process being done (vid 1) and locked down (vid 2) but only for a single GPU setup.

(2) Then read these two threads on Overclocking Multiple Nvidia GPUs under Linux - (a) https://foldingforum.org/viewtopic.php?f=80&t=29726 ; and (b) https://unix.stackexchange.com/question ... tions-cuda .

(3) Finally, if you have more questions feel free to Google this - "multi nvidia gpu overclocking for computations" - to study a couple of tries by someone else or you can just ask me your specific question or other matter of concern about overclocking GPUs under Linux.

P. S. There aren't yet any apps like AfterBurner for Linux.
Because I have 180+ GPU processers in 16 tweaked/multiOS systems - Character limit prevents detailed stats.
User avatar
Tutor
Licensed Customer
Posts: 531
Joined: Tue Nov 20, 2012 2:57 pm
Location: Suburb of Birmingham, AL - Home of the Birmingham Civil Rights Institute

Seekerfinder wrote: ...

Which Xeons did you go for? We're looking at E5 2687W V4's for our next build. But that's because we also want the machine to act as a workstation (3ghz+).

Seeker
Seeker,

Will the workstation be tasked with any CPU rendering? Have you ruled out the E5 2695 V4s [ http://www.cpu-world.com/Compare/515/In ... 95_v4.html ]? How about ES chips - http://www.ebay.com/itm/Intel-Xeon-E5-2 ... Sw32lY2oiL or http://www.ebay.com/itm/Intel-Xeon-E5-2 ... 2620269793 . ? What motherboard will you be using?
Last edited by Tutor on Tue Jul 18, 2017 3:48 pm, edited 1 time in total.
Because I have 180+ GPU processers in 16 tweaked/multiOS systems - Character limit prevents detailed stats.
User avatar
Seekerfinder
Licensed Customer
Posts: 1600
Joined: Tue Jan 04, 2011 11:34 am

Tutor wrote:
Seekerfinder wrote: ...

Which Xeons did you go for? We're looking at E5 2687W V4's for our next build. But that's because we also want the machine to act as a workstation (3ghz+).

Seeker
Seeker,

Will the workstation be tasked with any CPU rendering? Have you ruled out the E5 2695 V4s [ http://www.cpu-world.com/Compare/515/In ... 95_v4.html ].
Hey Tutor,
Long time no speak. Good to see you're still being your usual helpful self around here and I appreciate the interest. I think I have now looked at all the Xeon options - Intel loves confusing people it seems! The E5-2687w seems to offer a great balance between core-count and frequency.

The machine will be an all-rounder. We use Revit extensively in the office as well as point cloud scanning - those things thrive on Hz and cores respectively. In addition, the machine will be used for video & graphics editing and also other renderers like Keyshot, which is CPU based. We have other machines that would be more focused (and less expensive!) but this will be our powerhouse. There were some compelling offerings on the i7 side but we need those 80 PCIe lanes since it will house 7 GPU's.

Best,
Seeker
Win 8(64) | P9X79-E WS | i7-3930K | 32GB | GTX Titan & GTX 780Ti | SketchUP | Revit | Beta tester for Revit & Sketchup plugins for Octane
User avatar
Tutor
Licensed Customer
Posts: 531
Joined: Tue Nov 20, 2012 2:57 pm
Location: Suburb of Birmingham, AL - Home of the Birmingham Civil Rights Institute

Seeker, please note my additional questions.
Because I have 180+ GPU processers in 16 tweaked/multiOS systems - Character limit prevents detailed stats.
User avatar
Seekerfinder
Licensed Customer
Posts: 1600
Joined: Tue Jan 04, 2011 11:34 am

Tutor wrote:How about ES chips - http://www.ebay.com/itm/Intel-Xeon-E5-2 ... Sw32lY2oiL or http://www.ebay.com/itm/Intel-Xeon-E5-2 ... 2620269793 . ? What motherboard will you be using?
Those are great options but, the workstation cum server dilemma is getting Hz. We need at least 3GHz for a decent response in some of these apps.

After lots of research, ASrock Rack EP2C612 WS is currently the front runner.
Win 8(64) | P9X79-E WS | i7-3930K | 32GB | GTX Titan & GTX 780Ti | SketchUP | Revit | Beta tester for Revit & Sketchup plugins for Octane
User avatar
smicha
Licensed Customer
Posts: 3151
Joined: Wed Sep 21, 2011 4:13 pm
Location: Warsaw, Poland

Tutor,

What shall I run to run Octane slave on linux (ubuntu)?
3090, Titan, Quadro, Xeon Scalable Supermicro, 768GB RAM; Sketchup Pro, Classical Architecture.
Custom alloy powder coated laser cut cases, Autodesk metal-sheet 3D modelling.
build-log http://render.otoy.com/forum/viewtopic.php?f=9&t=42540
User avatar
Tutor
Licensed Customer
Posts: 531
Joined: Tue Nov 20, 2012 2:57 pm
Location: Suburb of Birmingham, AL - Home of the Birmingham Civil Rights Institute

Often times, how/when one starts a journey takes one to sites unseen by another who started his/her journey from a different place/time, but their fates are that their paths converge. That isn't intrinsically either good or bad, but it does limit the sites one sees and one's ability to imagine with perfection the sites seen by others who have traveled different routes/at different times. To me, the good and the bad of Linux is that there are multiple parent versions and hordes of children springing from each parent group which makes trouble shooting a ...... It makes for nightmares for early Linux explorers because it's usually roadmapless, i.e., there aren't then step by step tutorials, e.g., on how to run Octane slave on a particular Linux varient. For Ubuntu 14.14 LTS, if there's an extant roadmap, it's great at hiding - go seek.
smicha wrote:Tutor,

What shall I run to run Octane slave on linux (ubuntu)?
smicha wrote:Guys,

I just built 11 1080ti machine and it runs so well on linux Ubuntu 14.04 LTS.

I need your help - I am very new to linux - how to run Octane slave. When I run .sh file it asks me for port - 48000 as defaults - but then strange info comes... Is there any step by step tutorial of how to run Octane slave on linux?
Smicha,

When I started using computers back in the mid 1980's, I began as a Mac Man and later in the 1980's became a Mac/Amiga Man and later in the 1990's became a Mac/Amiga/Windows Man and still am to this very day. Yes, I still use my ancient Video Toasters. It wasn't until the mid-2000s that I began exploring Linux, and later in the post 2010+ era begin settling upon using Linux Mint as my favorit brand of Linux. I have never used the pure brand Ubuntu. I have never foresaken the Mac as my favorite system. Before the release of Octane's net render functionality, I relied upon discreet network render applications and those made a part of other applications, such as C4d. It wasn't until about 2013 that heard about OctaneRenderer. It wasn't until 2016 that I started using Octane's netrender application and because of my partiality for the Mac, I've always tried to make my Mac the master for my render farm (except for where the 3d application required me to use Windows as the OS). My renderfarm now consists of many systems running MacOS, Linux Mint, Windows 7 and Server. So, until you recently asked me the question: "What shall I run to run Octane slave on linux (ubuntu)?", I would have thought about this discussion that I ran across earlier - viewtopic.php?f=14&t=41153&p=195020&hil ... 3F#p195020 and answered " just run the Octane slave on the Ubuntu system. But now I know that would't have been a wise response on my part because it appears that you've already done that. I now believe that I need additional information about where you're fully at on your journey to UbuntuVille and how you want to get there.

So, what system OS(s) will you be running on this network? Which system OS(s) do you intend to be on the slaves and which OS(s) will you be running on the master? Theoretically, it shouldn't even matter - at least in a perfect world, but we all know that hasn't ever been yet achieved (nor do I expect it ever will be). So given the world's present, specific imperfections as they relate to Octane netrender installation on various flavors of Linux and other OSs, I also have to ask you to please provide me with that "strange info [output]" so that I can better try to analyze the problem.

Your friend to the end of time - Tutor

P.S. You might want to consider this too - https://askubuntu.com/questions/230689/ ... local-file
Because I have 180+ GPU processers in 16 tweaked/multiOS systems - Character limit prevents detailed stats.
User avatar
smicha
Licensed Customer
Posts: 3151
Joined: Wed Sep 21, 2011 4:13 pm
Location: Warsaw, Poland

Tutor, you are my light.

I will install Linux on my another machine and will report all back soon.

Yes - Linux is the slave with 11 gpus, win 7 is the master.

PS. I love linux - just 1 week of using it and am giving up windows wherever I can.
3090, Titan, Quadro, Xeon Scalable Supermicro, 768GB RAM; Sketchup Pro, Classical Architecture.
Custom alloy powder coated laser cut cases, Autodesk metal-sheet 3D modelling.
build-log http://render.otoy.com/forum/viewtopic.php?f=9&t=42540
User avatar
Tutor
Licensed Customer
Posts: 531
Joined: Tue Nov 20, 2012 2:57 pm
Location: Suburb of Birmingham, AL - Home of the Birmingham Civil Rights Institute

My renderfarm is reminisent to Sherwood Forest because there's robbery occurring.

Smicha,

The reason why I went to the lengths that I did with my system history/preferences and highest preference for using my Macs as my workstations (1) was to show why I most often chose one of my Mac Pro(s) as the master and (2) why I run Octane netrender (which is to be installed on the master only (but then pilfers [after correct setup from the connected slave])) from on a Mac Pro as opposed to one of my Linux systems - the slave. When I need more compute performance than those setups provide, I can use a Windows system as master (to pair with Linux slaves, as well as the master of those Mac +/or Windows combos. Thus, this route doesn't require running Octane netrender from a Linux system as the controller of another Linux system. To me, Octane netrender is like a thief that steals the GPU compute capabilty (and only that resource) from the slave(s) designated to be pillaged as chosen during setup on the master. So, my Macs and Windows systems (which can't even hold a flickering candle to the GPU compute capabilty of my Linux systems - because of the greater Input/output capabilty Linux holds over Mac & Windows OS ) siphon from the IO rich Linux systems by making it appear that the Mac/Windows system also owns the Linux system's GPU compute capabilities, i. e. appearing to give the total credit for the rendering out[put/come] to themselves - while yet they remain truely the poorer of the participants.

Nothing that I've reported, above, was intended to negate the fact that I have gotten one system while running Linux (and on another occasion Windows) to steal GPU compute capabilty from another Linux system using netrender. I just don' t see any reason to do so on the vast majority of my projects because it's wasteful as I might greatly exceed the 20 GPU per license limit as appears to be applicable when running netrender. I use the term "might" because I have seven Octane licenses and thus - to my feeble brain - I should be allowed to run at least 140 of my 140+ GPU processors at an Octane netrender block party so long as I can wedge groups of 20 of my GPU processors into each of seven systems. But, that's another gripe of mine for another day of fulllout ranting.
Because I have 180+ GPU processers in 16 tweaked/multiOS systems - Character limit prevents detailed stats.
Post Reply

Return to “Off Topic Forum”