Best Practices For Building A Multiple GPU System

[glimpse]

There's an 8xTitanX benched on OctaneBench at 1,007 Pts. Who are you?

straight from result page, it seems like someOne with server box or splitters & without tuning too much =) as 1007 is something You'd easily get on air without tuning =)

Anyway, so cool to see those results breaking 1000!

[Tutor]

There's an 8xTitanX benched on OctaneBench at 1,007 Pts. Who are you?

straight from result page, it seems like someOne with server box or splitters & without tuning too much =) as 1007 is something You'd easily get on air without tuning =) ... .

I agree - 1007 / 8 = 125.875 Pts per GPU. I would have estimated a score for 8xTitanXs, even on air, to be a bit higher with minimal tweaking and adequate cooling. Since its at the average score per GPU, however, that's why I'm skipping Maxwell and waiting for Pascal.

[sadece]

that's why I'm skipping Maxwell and waiting for Pascal.

Hi,
What do you expect from Pascal. i'm about to build a new system with 3x X or 980ti on a X99 E WS but never been sure since i've heard about the Pascal first time. And now more complicated. Even though, Nvidia says Pascal is 10x faster than Maxwell, i do not believe that. OK; 1Tb bandwith speed and some other improvements etc.. compared to Maxwell. BUT! what about the navite gpu computing performance? this is what makes me curious wit the new Pascal.

[glimpse]

Sadece, are You going to bild workstation or render node?

For node that motherboard + CPU & DDR4 RAM is too expensive combination =) but for main workstation might be justifieable if You plan to have fast (NVMe) storage & would like Your system to last longer (few GPU upgrades).

NVidia hasn't released any more information, just about speed, efficiency..however if they make a giant leap, consider to pay premium..one thing for sure, this architecture goes to data center first. Nvlink (together with better eficiency) is going to be very welcome feature out there.

Nvidias move to release 12gb TitanX is something they made with cool face considering their quadro has the same amount of vRam. My guess we'll see few more "upgrades" - small improvements & high-end quadros/teslas to be released.

If You need power badly, get 980Ti or TitanX they are here to stay..like original Titan..however with furry X closing the gap it's very likely TeamGreen is going to release dual GPU card (though I haven't heard any roumors about that =) but that's my guess..(how about 300points on single watercooled card sounds? =)

[sadece]

It's going to be a workstation and my current in the signature will not be a bad node which hits 170 score on air. i'm against everthing that consumes lanes except gpus so, fast storage ain't a need atm. but "last longer" is the key. anyway i'm open to any advise. i would drop you a pm just before my NY vacation end of Sept. where i'm going to buy components from. the prices almost 1/2 compared to my country. Btw, Titan series sounds me a trick of marketing. my first card was a Titan and 2nd one is 780 strix which hits same score but lower price. 300pts on a single card ? I'd wait for it from night to morning in front of the store just like iphone lovers do

[Tutor]

that's why I'm skipping Maxwell and waiting for Pascal.

Hi,
What do you expect from Pascal. i'm about to build a new system with 3x X or 980ti on a X99 E WS but never been sure since i've heard about the Pascal first time. And now more complicated. Even though, Nvidia says Pascal is 10x faster than Maxwell, i do not believe that. OK; 1Tb bandwith speed and some other improvements etc.. compared to Maxwell. BUT! what about the navite gpu computing performance? this is what makes me curious wit the new Pascal.

Have you ever seen a high Octane(render) burn ? If not, then get ready to see many of them soon- as early as Q2 2016.

I expect, if not this, something very close to it because last month NVIDIA’s Pascal GP100 chip began being taped out on TSMC’s 16nm FinFET process.

Here's a few highlights:

1) More than twice the no. of transistors (17 Billion) of Maxwell (8 Billion) and AMD’s Fiji (8.9 Billion).

2) Compute focused GPUs {whereas Maxwells aren't}. Pascals will support FP16, FP32 and FP64.

3) 4096bit memory interface - 1 TB/s bandwidth (vs. 334GB for 980Ti and 512GB for AMD's Fury), with 16G - 32G Vram.

4) NVLINK will allow several GPUs to be connected in parallel, whether in SLI for gaming or for professional usage, yielding 5-12 times bandwidth of regular PCIe connections [great for multi GPU environments}.

5) 40% higher speed and 60% power savings, e.g., RD Titan X has base clock of 1,000 MHz and TDP of 250W. Now just apply that percentage increase in speed and that percentage reduction in power to a 16G compute focused GPU with 1 TB/s memory bandwidth.

6) Will support 4k and 8k panels.

[sources: http://wccftech.com/nvidia-pascal-gpu-1 ... s-in-2016/ and
http://www.fudzilla.com/news/graphics/3 ... ransistors ]

P.S. (1) That's why I'm skipping Maxwells and planning to dedicate a single system (that supports 12+ GPUs) to mainly Pascal and Volta GPUs. (2) 2017 - Volta GPU to follow, fueling 150 PetaFlop systems in the HPC market.

[sadece]

Thank you Tutor for the feedback. That really makes sense.

[Notiusweb]

Hello! I see crashing behavior with Octane Bench (current using the 2.17 demo) where higher overclocking stunts the run of the benchmark. It goes through its process and then just stops, sometimes within 10 seconds of the run. What is interesting is that the further away the overclock is from the stock profile, the earlier the run halts. Once it halts, I can then see the cards in MSI afterburner tapering down as if the render has ended. The PC itself stays on normal, and I have to close the bench out. If I run overclock profiles closer to stock, I never encounter this issue.

For example, I can run a more than modest overclocking profile through Afterburner which crashes the bench, but then can immediately go on to use this same profile in the Octane Render standalone, or plugin that I have, and see an increase of ~100 Ms/sec on the scene with the overclock profile vs the stock profile.

I don't know if this is a limitation of a multi-GPU test, or if the test doesn't know how to gauge/measure certain overclocking thresholds. But anyway, I can realize the effects as I am doing my rendering, but just can't measure them through bench. Just throwing it out there in case anyone ever sees the bench process stopping, but PC remains active, it might just be the program itself, and not anything wrong with the GPU, power, etc.

[Tutor]

Running Linux to maximize working GPU count may have a hidden tradeoff for some. Since neither EVGA's Precision X nor MSI's Afterburner yet support Linux, only Nvidia's overclocking scheme (which uses bit adjustment) which I mention in an earlier post in this thread works for overclocking the GPUs. However to date, Nvidia's overclocking scheme is currently only designed to assist gamers since it allows only GPUs that are attached to displays to be overclocked.

[Notiusweb]

Hello!

Continued messing with BIOS and have been able to successfully boot into Win 7 64 Bit with 13 GPU cores on the ASRock X79 Extreme 11.
Main Display Titan X direct on Mobo, 6 Titan Z on risers (3 the USB 3.0 PCIE risers/3 on the Amfeltec 4-way GPU Cluster).

NVid Ctrl Panel5.jpg

OctaneRender6.jpg

Photo Aug 31, 1 53 05 AM.jpg

Once I got 11th GPU working I added another Z as I have been having success with these, giving me the 12th and 13th GPU cores (I considered 2 more Titan Xs, but felt I was capped at 6GB anyway...).
I attempted to post scores on Octane Bench however for some reason they are not uploading (I started a separate post to see if anyone could share thoughts on that.) It does let you select the full 13 GPU on Octane Bench, but it seems the scores match as if it were only 12. As such I am inclined to believe that it is only 12 that are being run in the benchmark even when you select 13.
I also put in a request to see if MSI Afterburner could allow up to and including 13 GPUs via its UI.

I have the Creative Soundblaster X-Fi soundcard still running for music production on my slot Lane 7, and both USB 2.0 and 3.0 ports are active. I have been using my old 660 Ti as the testing card for all this, contemplating that further experimenting may get 14 GPU. To summarize, I have a 1500 W for the Mobo, then 2 separate 1600 W EVGA external power supplies. One is extended to a separate electrical circuit via a 14 gauge extension chord.

Imagining the glory of 12 Titan Xs...then imagining 12 'Pascals'...