10-GPU SuperServer by SuperMicro

Wed Apr 05, 2017 11:41 am

Hey all, new poster here. I help with several sites that run various rendering software but we're fairly new to Octane.

We have just installed a SuperMicro TRT2, a 10-GPU-capable monstrosity of a server with the intention of using it primarily for Octane renders. Right now we have 4x GTX 1080 FE and 4x 980ti's in it. It's also running an NVME boot drive, has a 20gigabit connection to our NAS, and two 3.5ghz hex-core Xeons. We plan on adding 2 more 980tis to bring the total GPU count to 10 tomorrow.

We bought the thing with Windows 10 but it would not boot or run with the Nvidia drivers installed at all, so that fell over. Windows 8.1 Pro runs fine on it. However I quickly learned about the Windows limitation of only being able to initialize 32 display heads. With the onboard video enabled (for IPMI purposes) that means we've been unable to get all 8 cards recognized in Windows. Tomorrow we're going to disable the onboard video to confirm all 8 cards can be initialized in Windows and recognized.

And after that we're going to put Linux on (planning on Ubuntu 14.04 because that's a requirement of one of the other renderers we're going to possibly be using).

What I want is recommendations and any guidance you can muster around getting this many GPUs running efficiently on a single chassis - we've seen pretty miserable scaling going from the 4th to 5th to 6th+ cards, with minimal gains by adding more cards. I'm hoping this is due to some kind of Windows limitation which will magically disappear with Linux (:

Thanks

Mon Apr 10, 2017 4:09 pm

I have been thinking about these server boards but ended up building a 4x system for a test. 4x 1080Ti runs fine with Windows 7.

I thought Octane scales linearly with the number of GPUs? Why not build a couple of 4x systems and just use as slaves? I haven't tried it yet so anyone who has experiences should chime in.

8-10x system versus multiple 4x systems as slaves.

Good luck and keep us posted.

Mon Apr 10, 2017 4:39 pm

tangelo-otoy wrote:
What I want is recommendations and any guidance you can muster around getting this many GPUs running efficiently on a single chassis - we've seen pretty miserable scaling going from the 4th to 5th to 6th+ cards, with minimal gains by adding more cards. I'm hoping this is due to some kind of Windows limitation which will magically disappear with Linux (:

Thanks

GHM, that shouldn't be the case.. windows here have nothing to do with performance of OctaneRender as far as I know.. but heat build up might have reduced some clockspeeds of those GPUs.

The best would be to fire up OctanaBench monitoring all GPUs & see what clockspeeds they run. Also what CPU is doing, since some parts of are not multi-threaded. Considering You have pretty slow single core speed that might be one of the things that hold things back a bit (but that's just a guess, based on what I've already noticed).

feel free to post some print-screens (made while running OctaneBench) if You would find some help being useful ;)

Tue Apr 11, 2017 8:52 pm

Hey all,

I thought Octane scales linearly with the number of GPUs? Why not build a couple of 4x systems and just use as slaves? I haven't tried it yet so anyone who has experiences should chime in.

I'm not the Octane user but I believe this has to do with rendering large jobs quickly; perhaps you can only submit one job to one slave, and the power of that slave determines how quickly the job is finished? For individual large jobs having more power in one chassis is better?

GHM, that shouldn't be the case.. windows here have nothing to do with performance of OctaneRender as far as I know.. but heat build up might have reduced some clockspeeds of those GPUs.

We only started seeing scaling degradation with scaling past 3 cards. Have you run more than that in a single chassis and seen perfect scaling?

Heat was definitely not a problem here as we were only running renders that took the machine about 2 minutes to run. The 1080s are also using Founders Edition shrouds backed by system fans that would push the server out of the rack if it weren't bolted in.

We have now got Linux running on it and seeing about 84% scaling efficiency to 10 cards, which is heaps better than we were seeing with Windows 8.1.

However now I have naother question - how do I get the octane daemon to run in Linux without having to log in on every boot? Can I save the credentials? There's no documentation I've been able to find on running Octane in Linux.

Thanks,

~Micah

Tue Apr 11, 2017 9:59 pm

tangelo-otoy wrote:how do I get the octane daemon to run in Linux without having to log in on every boot? Can I save the credentials? There's no documentation I've been able to find on running Octane in Linux.

There's nothing specific to Linux in terms of authentication. Your session should persist even after rebooting your machine. Do you see any error message during authentication? Maybe Octane is unable to save the credentials for some reason and that would force you to re-sign in every time.

Fri Apr 21, 2017 10:20 am

No, there is no error message on launching the daemon run script, it just asks for login credentials every time. I can do some permissions checking. Thanks.

Fri Apr 21, 2017 10:46 am

@glimpse I don't believe octanebench works with pascal cards. Recommendations?

Someone also said something about using TCC drivers to get around the windows limitations. Anyone have experience with those drivers and would they work in this scenario to help get all 10 cards running under Windows?

Sun Apr 23, 2017 6:05 am

1080ti & other Pascal cards works on "hacked" build - let me find You a link.

Again, most likelly it's not windows, but rather hardware-bios issue.

Sun Apr 23, 2017 6:19 am

From here You can download OctaneBench, under picture in first post You will find a link ;)
viewtopic.php?f=9&t=56108&hilit=1080#p287699

Tue Jan 08, 2019 9:19 pm

tangelo-otoy wrote:Hey all, new poster here. I help with several sites that run various rendering software but we're fairly new to Octane.

We have just installed a SuperMicro TRT2, a 10-GPU-capable monstrosity of a server with the intention of using it primarily for Octane renders. Right now we have 4x GTX 1080 FE and 4x 980ti's in it. It's also running an NVME boot drive, has a 20gigabit connection to our NAS, and two 3.5ghz hex-core Xeons. We plan on adding 2 more 980tis to bring the total GPU count to 10 tomorrow.

We bought the thing with Windows 10 but it would not boot or run with the Nvidia drivers installed at all, so that fell over. Windows 8.1 Pro runs fine on it. However I quickly learned about the Windows limitation of only being able to initialize 32 display heads. With the onboard video enabled (for IPMI purposes) that means we've been unable to get all 8 cards recognized in Windows. Tomorrow we're going to disable the onboard video to confirm all 8 cards can be initialized in Windows and recognized.

And after that we're going to put Linux on (planning on Ubuntu 14.04 because that's a requirement of one of the other renderers we're going to possibly be using).

What I want is recommendations and any guidance you can muster around getting this many GPUs running efficiently on a single chassis - we've seen pretty miserable scaling going from the 4th to 5th to 6th+ cards, with minimal gains by adding more cards. I'm hoping this is due to some kind of Windows limitation which will magically disappear with Linux (:

Thanks

Hi How do you like your 8 GPU server? We were planning to get the same one. Were you able to run it on windows 10 Thanks