Best Practices For Building A Multiple GPU System

[Notiusweb]

Hi,

We do not have the same scene, same resolution, and same settings. Send me your red car also.
But we know that v2 never freeze. V3 adds more transfer to the system, perhaps something to dig on the 1X PCie V2 or V1 or V3. My splitter 1 to 4 seems to be Gen 1 (bandwith of 2.5 GBits/s). The splitter 1 to 3 is specifed at Gen 2.2 (5 Gbit/sec)). and the cluster is Gen 2.2 (5 Gbit/sec).

Yeah, I agree, that would be interesting to see, however in my conversations with Amfeltec they seemed to view the Cluster, 3 and 4-way Splitter, and PCI Express Expansion Backplane as compliant with Generation 2.
I'm gonna guess there would be no difference between the 3 and 4-way splitter.

As far as Generation mode tested, I did try myself setting at Gen 1, Gen 2, and Gen 3 via BIOS, it still did not prevent freezing on the Expansion Cluster.
I was able to switch PCIE Generation in BIOS.

[itou31]

I test your car scene and my splitter freeze at 2700 x 2700 on normal camera, and instant free when I set baking cam.
I have played with TDrDdiDelay, DCI/TimeOut ... but same issue.
Test with 362.00 drivers and windows 8.1
Set also in BIOS PCIe GEN2.

[Notiusweb]

I test your car scene and my splitter freeze at 2700 x 2700 on normal camera, and instant free when I set baking cam.
I have played with TDrDdiDelay, DCI/TimeOut ... but same issue.
Test with 362.00 drivers and windows 8.1
Set also in BIOS PCIe GEN2.

Itou31, sorry to crash your PC with the red car.
I guess this is good in a way because it means that if a solution comes for one of us, it will be a solution for the other!
You are actually the first person to have the scene freeze, and Tutor was also using the 4-way splitter, may be a clue for us to find.

Speaking of which, Itou31, I really wonder why the USB Riser is failing for you but not for me. What is weird, is that they only worked when the Amfeltec is unpowered and not active at all.
This is the one I have, is yours the same, different? This could yield a huge clue too...
(it says no longer available...figures, so F'ing typical....but there are others actually that look the same, molex powered, etc.)

http://www.amazon.com/gp/product/B00IKT ... detailpage



Tutor, just double checking, you ran my file in the baking cam mode, where it turns to polygons, right?
It looked like this?

Car2.jpg

And you set to to a high 3840 x 2160 resolution, right?
And you ate a bowl of the most delicious and healthy cereal of all time, Cinnamon Toast Crunch, while rendering, right?

[Tutor]

I test your car scene and my splitter freeze at 2700 x 2700 on normal camera, and instant free when I set baking cam.
I have played with TDrDdiDelay, DCI/TimeOut ... but same issue.
Test with 362.00 drivers and windows 8.1
Set also in BIOS PCIe GEN2.

Itou31, sorry to crash your PC with the red car.
I guess this is good in a way because it means that if a solution comes for one of us, it will be a solution for the other!
You are actually the first person to have the scene freeze, and Tutor was also using the 4-way splitter, may be a clue for us to find.

Speaking of which, Itou31, I really wonder why the USB Riser is failing for you but not for me. What is weird, is that they only worked when the Amfeltec is unpowered and not active at all.
This is the one I have, is yours the same, different? This could yield a huge clue too...
(it says no longer available...figures, so F'ing typical....but there are others actually that look the same, molex powered, etc.)

http://www.amazon.com/gp/product/B00IKT ... detailpage

I've read all of the intervening posts, but like I said earlier: "Of course, that doesn't rule out a defect (or other causation) in any particular piece piece of hardware that you're using." I don't have Amfeltec's Cluster, but I do have Amfeltec's 4 port splitters [The 3 port splitters appear to differ only in the number of ports - 3 vs. 4]. Not all x1, x4, x 8, and x16 powered and non-powered risers are the same. Variables such as the quality/suitablility of parts and quality of manufacture/assembly may vary. For example, all of my risers are molex powered. Others could have risers that are USB or SATA powered. Not all powering methods are equal for a particular use and cabling matters too. See, e.g., (1) http://www.tomshardware.com/forum/35594 ... -connector and (2) https://en.wikipedia.org/wiki/DC_connector .

BTW - I tested with 362.00 drivers and I used bios setting of PCIe GEN3.

Tutor, just double checking, you ran my file in the baking cam mode, where it turns to polygons, right?
It looked like this?

Car2.jpg

And you set to to a high 3840 x 2160 resolution, right?

Baking cam mode was on and resolution was set to 3840 x 2160.

And you ate a bowl of the most delicious and healthy cereal of all time, Cinnamon Toast Crunch, while rendering, right?

I didn't eat a bowl of Cinnanmon Toast Crunch. I cherish higher fiber and higher vitamin, mineral and protein content. I seldom consume meats other than seafoods. Before I performed the tests, I had a glass ( 8-10 ounces) of my juice with breakfast. I drink it twice daily, so I have it with dinner also. This drink consists of, at the very least, a blend of tomato/orange/cranberry & acai juices + protein powder + corn + soy beans + oats + barley + chia seeds + pumpkin seeds + flax seeds* + spelt + southern wheat* + quinoa + kamut khoransan & amaranth wheat + rice + hemp hearts + raisins + prunes + apple* + plum* + pear* + peach* + apricot + banana + concord & muscadine grapes (seeds included)* + cherries + figs (five varieties)* + dried dates + blackberries* + strawberries* + blueberries* + coconut + mango + pineapple* + kiwi + dragon fruit + lemon* + lime* + goji berries + seabuckthorn + black currants + mangosteen + noni berries + yumberries + raspberries* + pomegranate* + bilberries + pomelo + grapefruit + honeydew*/water-*/crenshaw* & cantalope* melons + mango + papaya + almonds** + hazel nuts** + peanuts + chestnuts** + pecans** + walnuts** + Brazil nuts + broccoli + Brussels sprouts + green algae + seaweed - cauliflower + artichokes hearts + beet* + eggplant + potatoes (both sweet and white* varieties)* + onion* + garlic* + brussels sprouts + okra* + cucumber*+ squash* + pepper* + radish + olive + turmeric root + carrot + peas (usually at least 4 different varieties)** and beans (usually at least fifteen different varieties)** + lettuce/turnip*/collard*/spinach/cabbage/kale and dandelion leaves.* I also add other fruits and vegetables as I can find them for sale at reasonable prices. And yes, I make this drink taste good; so good, in fact, that little children love to drink it.

N.B. Whenever possible, I use juice substitutes for some of the listed fruits and vegetables and packaged cereals for many of the listed grains. Also, for the vast majority of fruits and vegetables I use the whole plant - seeds and skins/rinds included, but I don't consume parts such as tree bark, coconut shells or peach seeds that are too hard to blend or are harmful when consumed in large quantities (such as apple seeds).

* denotes that I grow at least half of the fruit or vegetable myself.
•• denotes that I've already planted plants/trees to produce my own.

[Notiusweb]

Tutor, speaking of multiple ingredients, how many GPUs have you tried on V3 Alpha simultaneously.
I can use my 13 (12z and 1x), but I know you had more. And I believe the limit is 20 (? I think). Anyway, the V3 Alpha lets you go beyond 12, so check it out if you hadn't yet!

And boast about it too. There's no shame in having GPUs.

[Tutor]

Tutor, speaking of multiple ingredients, how many GPUs have you tried on V3 Alpha simultaneously.
I can use my 13 (12z and 1x), but I know you had more. And I believe the limit is 20 (? I think). Anyway, the V3 Alpha lets you go beyond 12, so check it out if you hadn't yet!

And boast about it too. There's no shame in having GPUs.

This, in part, is one of the current license terms that I was greeted with when I went to the latest V3 Alpha download page and clicked the download package:
"YOU MAY NOT: ... . Use more than 12 GPU's." So I'm using other GPU renderers for the time being because for my current projects I need support for, at a minimum, sixteen GPUs for a number of my systems.

[smicha]

For those of you who uses or will use Asus X99 E WS motherboard. These bios settings give better stability especially in a case when all 4 GPUs are stressed fully and a cpu is under load (on auto settings I had some issues with windows when rendering on 4 GPUs)

MENU AiTweaker:
Ai Overclock Tuner: XMP
CPU strap: 100Mhz
PLL selection: LC PLL
BCLK freq.: 100
Asus multicore enhancement :disabled
Cpu core ration: sync all cores
1-core ratio limit 40 (for some mild OC)
Min and max cpu cache ratio: auto
Internal PLL overvoltage: Disabled
EPU power saving mode: Disabled

Submenu: external digi + power control:
CPU power phase control: extreme
CPU power duty control: extreme
CPU current capability: 130%

Submenu: internal CPU power man.:
enhanced intel speedstep techn.: Disabled
turbo mode: enabled

Back to Ai tweaker: CPU core volt. - my reads at above settings is 1.2V (tested for 3 days - stable), so one can input 1.2V or leave on auto - cpuz shows 1.2V constantly - some users report that for 4.5GHz 1.275V may be enough but this is dependent of how lucky you are on cpu quality.
CPU spread spectrum: disabled

MENU: advanced/cpu power manag. config.
enhanced intel...: disabled
turbo: enabled
c states : disabled

Cheers.

BTW with the above settings and OC on GPUS I managed to OC 4 980ti on water to score 570 (not increasing the voltage on gpus, 1.2V). So there is a chance to reach the 1000 score on 7 watercooled 980ti.

[Tutor]

Smicha,

Thanks for update and especially for the level of detail regarding settings.

Here's a Pascal GPU Update:

http://wccftech.com/nvidia-pascal-gefor ... gp104-gpu/

Launch May-June 2016
NVIDIA GeForce GTX 1080 (replaces GTX 980)
Graphics Core = GP104
Process Node = 16nm FinFET
Transistors & CUDA Cores = TBD
VRAM = 8 GB GDDR5/X
256-bit bus?
Power Connector = Single 8-Pin Power
TDP = < 225W
Display Connectors = 2x Display Port, 1x HDMI, 1x DVI

Launch Sept.- Dec. 2016
NVIDIA GP100 (replaces GTX TitanX/980 Ti)
Graphics Core = GP100
Process Node = 16nm FinFET
Transistors & CUDA Cores = TBD
VRAM = 16 GB HBM2
4,096-bit bus
Power Connector = 8+6 Pin Power
TDP = 250W
Display Connectors = TBD

Performance - Best Guesses To Date for High End GTX Pascal TIs and Titans
http://videocardz.com/58297/nvidia-pasc ... erformance

Either 6,144 CUDA cores clocked at 1,000 MHz or 5120 CUDA cores clocked at 1,200 MHz (But range projected from 5,120 to 7,680 CUDA cores and from 40 to 60 SMXs). So it would seem that we GPU renderers for the first time might see close to a real doubling of rendering performance per high end GTX single GPU processor comparators between succeeding generations. If the Pascal projections are near accurate, that would mean that 7 Pascal Tis might score close to 2,000 points using the current benchmark or 12 of them might score close to 3,410 points using that same benchmark. I'd love to have 12 dual GPU processor GTX Pascals and the ability to run them all under one Octane license (with a license limit of > 24 GPU processors).

[smicha]

Smicha,

Thanks for update and especially for the level of detail regarding settings.

Here's a Pascal GPU Update:

http://wccftech.com/nvidia-pascal-gefor ... gp104-gpu/

Launch May-June 2016
NVIDIA GeForce GTX 1080 (replaces GTX 980)
Graphics Core = GP104
Process Node = 16nm FinFET
Transistors & CUDA Cores = TBD
VRAM = 8 GB GDDR5/X
256-bit bus?
Power Connector = Single 8-Pin Power
TDP = < 225W
Display Connectors = 2x Display Port, 1x HDMI, 1x DVI

Launch Sept.- Dec. 2016
NVIDIA GP100 (replaces GTX TitanX/980 Ti)
Graphics Core = GP100
Process Node = 16nm FinFET
Transistors & CUDA Cores = TBD
VRAM = 16 GB HBM2
4,096-bit bus
Power Connector = 8+6 Pin Power
TDP = 250W
Display Connectors = TBD

Performance - Best Guesses To Date for High End GTX Pascal TIs and Titans
http://videocardz.com/58297/nvidia-pasc ... erformance

Either 6,144 CUDA cores clocked at 1,000 MHz or 5120 CUDA cores clocked at 1,200 MHz (But range projected from 5,120 to 7,680 CUDA cores and from 40 to 60 SMXs). So it would seem that we GPU renderers for the first time might see close to a real doubling of rendering performance per higher end GTX GPU comparators between succeeding generations. If the Pascal projections are near accurate, that would mean that 7 Pascal Tis might score close to 2,000 points using the current benchmark or 12 of them might score close to 3,410 points using that same benchmark.

That is what I read in the morning drinking coffee It would confirm my earlier calculations that we may witness even 2-3x speedup. Let's wait and see. Thank you!

PS Hope you saw it viewtopic.php?f=9&t=53004

[Tutor]

... .
PS Hope you saw it viewtopic.php?f=9&t=53004

Smicha,
Double thanks. I hadn't seen it before, but now having seen it, I'll repeat here my comment there:

"Exquisite build. It's breathlessly, magnificently powerful and beautiful." But that's not surprising to me because I know that you're the man blessed with awesome talents.