Best Practices For Building A Multiple GPU System

[Notiusweb]

Tom, were you referring to a physical ground wire here? That's interesting you ask that if that is what you meant, Amfeltec first asked me to attach a ground wire to the PSU, to the Amfeltec Chassis, and then to the PC chassis. I actually tried it 2x with a 10-guage ground wire, with no effect. When you find yourself doing unusual things to make something normal you start to feel like you have a problem with that arrangement.

Yeah, I was refering to this. A while ago one Guy on Fb group was in panic as He couldn't even start computer (minig rig like setup) with more GPUs it wasn't moving anywhere not booting at all (he was using splitter from Amfeltec). I was like ready to send my splitter (as I had working tested unit sitting unused), but then Amfeltec support suggested to ground things up & that solved problems. So I was thinkin' maybe that (missing ground somewhere) has anything to do with stability in Your case as well.

Thanks Tom, I appreciate your suggestion and the helpful guidance.
In my own conversations with Amfeltec they had that among their leading steps of debugging.

[Seekerfinder]

... .
We're going to try and do that at least to a degree with the GPU Turbine. Right now, we're at dual CPU and dual PSU.
Best,
Seeker

How is the Turbine arranged, does it connect externally to a motherboard like a GPU splitter with one cable connecting to a PCIe slot, or is the motherboard integrated into it structurally somehow, and it has GPUs independently connected to the motherboard?

Hi Notius,
The GPU Turbine is a mechanical solution. We have a large fan that cools the cluster-chamber, up to seven cards on air. We largely isolate the chamber in order to offer efficient linear airflow, focused on the GPU's only. In addition, it accommodates a second power supply. It is designed to work with a regular or server motherboard - we don't integrate one. Risers are therefore required. The unit is designed to be very flexible (very...). So the user could use all the components they already have and just add risers and a second power-supply. It's all on http://www.pixelredemption.com

It's evolved quite a bit from it's initial design. More soon...

Best,
Seeker

[Notiusweb]

Tutor, do you find it a best practice to invest in spare parts for your rigs ("live and let live") or do you put investment into newer parts for newer rigs and leave older ones to the test of time ("live and let die")?

Also, on an inverted note, reading discussions about the ORC (Octane Render Cloud) pricing, it is a funny thing. It's inevitable that it will be of value to those who haven't purchased a lot of GPU and rig hardware. For those of us who have though, then the value becomes related to how technologically powerful of a render farm you get. Right now in its early stages it seems to be modest (a $9.99 purchase = 800 bench for 1.5 hrs...I think) because we'd match or better that already with our own local rigs. But if it was like 3,000 bench, I'd feel like I would have to try it out.

[smicha]

Some short (but very important note) for asus x99-e ws mobos (or any other PLX mobos): before putting more than 4 gpus onto the mobo change 4G decoding to enabled. And never NEVER update bios with more than 4 GPUs on the mobo.

PS. If you see 00 code and James Bond with magic 7 is not appearing - your mobo is down. If you see non-zero codes and blue diodes blinking on the mobo but it restarts simply leave only 4 gpus on the mobo (1,3,5,7 slots) and change back 4G to enabled.

[Seekerfinder]

And never NEVER update bios with more than 4 GPUs on the mobo.

This one is new to me - thanks Smicha!
Pesky PLX's...

Seeker

[Tutor]

Tutor, do you find it a best practice to invest in spare parts for your rigs ("live and let live") or do you put investment into newer parts for newer rigs and leave older ones to the test of time ("live and let die")?

Question (instead of an answer): Is there a significant reason why I should invest in newer parts for newer rigs and leave my older ones to the test of time ("live and let die") so long as (1) newer GPU cards perform equally as well in those old timers, (2) CPUs aren't yet getting radically faster (I do perform CPU rendering) and (3) the old timers' ability to handle additional GPUs hasn't been exhausted?

Also, on an inverted note, reading discussions about the ORC (Octane Render Cloud) pricing, it is a funny thing. It's inevitable that it will be of value to those who haven't purchased a lot of GPU and rig hardware. For those of us who have though, then the value becomes related to how technologically powerful of a render farm you get. Right now in its early stages it seems to be modest (a $9.99 purchase = 800 bench for 1.5 hrs...I think) because we'd match or better that already with our own local rigs. But if it was like 3,000 bench, I'd feel like I would have to try it out.

ORC is arriving too late to have suited my past (and even present) needs and what I've done to satisfy those needs has made it less likely that I will often see value from ORC in the next few years, at least that's the way it seems presently; but things do change and appear to be changing faster so the future might dictate that I often supplement what I'm running at some particular point in the future. Thus, the then speed, costs, ease of use, security, reliability, etc. of services such ORC's at that future time vs. my gear then, the cost to augment it locally, and other comparative considerations would be factors that I'd assess.

[Notiusweb]

Question (instead of an answer): Is there a significant reason why I should invest in newer parts for newer rigs and leave my older ones to the test of time ("live and let die") so long as (1) newer GPU cards perform equally as well in those old timers, (2) CPUs aren't yet getting radically faster (I do perform CPU rendering) and (3) the old timers' ability to handle additional GPUs hasn't been exhausted?

Let me rephrase my question. I am assuming you are adding newer builds over time to meet consumer demand, so at default you will be required to put money towards a new system(s) at some future interval(s).
In between said intervals, you have the option to either purchase spare parts for your current rigs in case of a hardware issue, or hold off and keep money for future rig builds.
From your years of experience, have you found there is a scenario that is more statistically likely to occur:
(1) your rigs need a spare part often and you are glad you bought spare parts (both small and large cost, ie DRAM, Mobo, wires, PSUs, tubing, hard drives, etc)
(2) your rigs almost never needs spare parts, and/or need for repair/part replacement is not likely...money is better invested on future builds.

[Tutor]

Question (instead of an answer): Is there a significant reason why I should invest in newer parts for newer rigs and leave my older ones to the test of time ("live and let die") so long as (1) newer GPU cards perform equally as well in those old timers, (2) CPUs aren't yet getting radically faster (I do perform CPU rendering) and (3) the old timers' ability to handle additional GPUs hasn't been exhausted?

Let me rephrase my question. I am assuming you are adding newer builds over time to meet consumer demand, so at default you will be required to put money towards a new system(s) at some future interval(s).
In between said intervals, you have the option to either purchase spare parts for your current rigs in case of a hardware issue, or hold off and keep money for future rig builds.
From your years of experience, have you found there is a scenario that is more statistically likely to occur:
(1) your rigs need a spare part often and you are glad you bought spare parts (both small and large cost, ie DRAM, Mobo, wires, PSUs, tubing, hard drives, etc)
(2) your rigs almost never needs spare parts, and/or need for repair/part replacement is not likely...money is better invested on future builds.

Thanks for the clarification. My current rigs almost never need spare parts (in fact, excluding my Asus experiences, I can't remember any repairs ever having been needed), except for upgrades such as more powerful CPU(s) or to increase the GPU count or add more system memory, so the need for repair/parts replacement has luckily been nonexistent unless you meant to include upgrades. I do plan future Supermicro mega PCIe count builds until death does part rendering and me.

[Notiusweb]

So I don't know if you guys saw but Otoy did put forward a selectable 'Tone-mapping' option per device in V3 Alpha 10. What this allowed for was to select as few as only 1 GPU to process Post Processing, or baking cam. It was Post Processing, and baking cam mode, which both independently were making my Amfeltec cards crash. It caused a higher than stable Deffered Procedural Call ("DPC") latency between the Nvidia driver and the Direct X driver on my system. (353.62 through 364.72, and DX11)

So whereas if I have 6 GPU on Amfeltec (3 Titan Z), in 3.9 they were involved in Post Processing and/or Baking Cam processing, and they caused high latency communications in my PC, and they were crashing my PC.
But in 3.10, I can have one, or all, of my USB riser cards exclusively act as the active tone-mappers (both rendering and handling the Post Processing), while the 6 Amfeltec GPUs need only be used for rendering (non-active tone-mappers). Similarly, if I only assign tone-mapping to the USB riser cards when using baking cam mode, it will not crash, but will still have the rendering speed of all 12 GPU. This allows me to fully use V3 with Amfeltec! Although, I have not enabled the Amfeltec back into my system yet as the plugins do not have the Alpha 3.10 feature set yet. That may be ways off. But it was good to try and see it could work.

I have noticed now, even when only using my Titan X at 16x, that if I have a scene with a high resolution, let's say 1920 x 1080, and I have post processing on, the maneuverability of panning zooming and scaling gets choppier and slowed down. But once I turn Post Processing off, it gets much faster and smoother. I am curious if you all see this, or if it is only my system. In other words, on a high-res scene, do you notice the slowdown on maneuvering a scene with your mouse while post processing is enabled vs not enabled.

PS - Any more Pascalian rumours?

[Tutor]

... .

Thanks for update.

PS - Any more Pascalian rumours?

Yes. But, I'm not taking the bait. Check you PMs for a glimpse of what's to come & when.