12 GPU Limit

[glimpse]

Z do overheat. One fan for two two gpus. ..

what i mean is that if you have one pwr box ok, but stacked? no way, unless you watercool of course.

Titan Z is not overHeating, it just keeps itself under predefined temperature target (as part of a plan for Boost 2 technology).

Titan Z however on render times drops it's speed from almost 1050MHz to 700MHz+ while TitanBlack drops not so much - from the same starting point to somewhat 800-900MHz (depending on airflow). Both Z & Black are tested in the same case (I have print screens of the my desktop now).

The average 780 or 980 has 3 fans per gpu.

reference 770,780(TI), Titan(Black), 970 & 980 has single fan too..
only non reference coolers has 2-3 fans..-this is not perfect solution, considering the fact all the heat is circulating inside the case (in multi GPU, let's say 4 GPUrigs, non reference design coolers with those 3fans perform worse than reference design cooler with single fan, throwing heat out of the case).

So, fans are nothing to do with more efficient cooling in multi GPU environment - that's not what they were designed for in the first place - representatives from companies like ASUS do acknowledge this that reference coolers work better in 4way configurations, not long ago seen video with JJ talking about that =)

GPU PCBs tend also to bend from heating and cooling ( on and off computer ) unless there is a backplate. The board gets bent, soldering points get damaged.

bending PCB is a marketing trick to sell You a piece of worthless metal for 10-30 dollars - it has little to no effect to GPU. The only benefit is when Your OCed ram sits on the other side of GPU - then backplate acts as a giant heat-spreader.

However (again) in multi GPU configurations, especially with non reference designed coolers (2-3fans) those backplates obstruct airflow & make more damage that giving something good - as temperatures rises too much because of insufficient airflow..

so let's get back to the topic, TitanZ is not overheating, it keeps itself under control. If You want to run it's GPUs on the higher clock speeds get more airflow into case, raise fans speed & lift the temperature target (it will get more noisy, but You keep higher performance).
Or go water-cooling, leaving noice behind & enjoy high clock rates =)

[smicha]

Z do overheat. One fan for two two gpus. ..

what i mean is that if you have one pwr box ok, but stacked? no way, unless you watercool of course.

Titan Z is not overHeating, it just keeps itself under predefined temperature target (as part of a plan for Boost 2 technology).

Titan Z however on render times drops it's speed from almost 1050MHz to 700MHz+ while TitanBlack drops not so much - from the same starting point to somewhat 800-900MHz (depending on airflow). Both Z & Black are tested in the same case (I have print screens of the my desktop now).

The average 780 or 980 has 3 fans per gpu.

reference 770,780(TI), Titan(Black), 970 & 980 has single fan too..
only non reference coolers has 2-3 fans..-this is not perfect solution, considering the fact all the heat is circulating inside the case (in multi GPU, let's say 4 GPUrigs, non reference design coolers with those 3fans perform worse than reference design cooler with single fan, throwing heat out of the case).

So, fans are nothing to do with more efficient cooling in multi GPU environment - that's not what they were designed for in the first place - representatives from companies like ASUS do acknowledge this that reference coolers work better in 4way configurations, not long ago seen video with JJ talking about that =)

GPU PCBs tend also to bend from heating and cooling ( on and off computer ) unless there is a backplate. The board gets bent, soldering points get damaged.

bending PCB is a marketing trick to sell You a piece of worthless metal for 10-30 dollars - it has little to no effect to GPU. The only benefit is when Your OCed ram sits on the other side of GPU - then backplate acts as a giant heat-spreader.

However (again) in multi GPU configurations, especially with non reference designed coolers (2-3fans) those backplates obstruct airflow & make more damage that giving something good - as temperatures rises too much because of insufficient airflow..

so let's get back to the topic, TitanZ is not overheating, it keeps itself under control. If You want to run it's GPUs on the higher clock speeds get more airflow into case, raise fans speed & lift the temperature target (it will get more noisy, but You keep higher performance).
Or go water-cooling, leaving noice behind & enjoy high clock rates =)

wise

[abstractmedia]

Powerful build man!

16 GPUs would be amazing! I have been maxed out for over a year now.

Abstract,
Put your rig in your sig... That way we can see what you got.

Seeker

It's changing to often to put it in a sig, Currently we are running 12 Titans, with some 970s for off renders.

[kraken]

Doing more investigation, only leaves me with more questions. I found a Tesla S1070 with 996 CUDAs, but 16 GBs of GDDR5 RAM and 4GPU's. It claims to do about 4 teraflops of computation...I sort have an idea what that means, but what I don't know is how does it stacks up against a Titan, or any powerful GTX? Sure the titan has more CUDAs but does it still out perform this Tesla and by how much/little? I'm also noticing, that some of my modeling programs are running OpenGL. (Does OpenGL use CUDA cores or just the GPU?...or am I misunderstanding the whole concept of what you need a lot of GDDR RAM in the first place?)

I'm trying to make a purchasing decision and have no idea how to figure out "bang for bucks" ratio.

Any help would be much appreciated.

Thanks

[glimpse]

Doing more investigation, only leaves me with more questions. I found a Tesla S1070 with 996 CUDAs, but 16 GBs of GDDR5 RAM and 4GPU's. It claims to do about 4 teraflops of computation...I sort have an idea what that means, but what I don't know is how does it stacks up against a Titan, or any powerful GTX? Sure the titan has more CUDAs but does it still out perform this Tesla and by how much/little? I'm also noticing, that some of my modeling programs are running OpenGL. (Does OpenGL use CUDA cores or just the GPU?...or am I misunderstanding the whole concept of what you need a lot of GDDR RAM in the first place?)

I'm trying to make a purchasing decision and have no idea how to figure out "bang for bucks" ratio.

Any help would be much appreciated.

Thanks

drop a line about Your needs, workflows & other things (including budget) - I'll give You some specific ideas based on Your case =)

Simply put, Octane relies on SP performance..if You don't need more than 6GB, then 7806GB will give the best value, then TitanZ & Finaly TitanBlack..- Quadros start make sense if You need more than 6GB & looking for other features like specific driver support & DP performance.

cheers

[kraken]

Thanks for the consultation.

I'm finding that animating and fx are more my interest. Most of the modeling software I use (and have looked into) run on CPU so there's not much I can expect to gain from any GPU upgrades. If I were simply interested in making killer static shots, then I would be happy with my GTX 690 and call it a day, but as it turns out I can't stop animating everything I model. I just love it. It's like breathing life into pixels. Of course, rendering can take days and I'm very impatient. I would like to have something that could generate a hi-res (2K) at a reasonable pace. One frame per minute would be acceptable (sure, real-time would be insane).

This is where I'm getting mixed signals. I see these Tesla selling for $n-Thousands, yet the hardware specs don't seem to compete with a Titan. SO, why are they so expensive? Surely they do something that a Titan can't or they wouldn't be selling for so much money. AND since I'm mainly interested in GPU rendering is it safe to assume that Octane really only cares about CUDA's or do the other factors (RAM, GPU, Bandwidth...) have any bearing?

Lets say someone can buy a GTX 690 or a Tesla S1070 for half the cost with the specific goal to render animations with Octane. Which choice is best?

Operating System: Windows 7 Ultimate 64-bit (6.1, Build 7601) Service Pack 1 (7601.win7sp1_gdr.141211-1742)
Language: English (Regional Setting: English)
System Manufacturer: INTEL
System Model: DZ77BH
BIOS: Default System BIOS
Processor: Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz (8 CPUs), ~3.4GHz
Memory: 32768MB RAM
Card name: NVIDIA GeForce GTX 690
Display Memory: 4004 MB

[glimpse]

kraken, Tesla Products are ment for server space - some of them have coolers that are designned for rack mounted cases with very strong airflow. The price is different, cos they have different target group, with specific needs..again as mentioned PRo line products have different drivers & ussully higher vRam counts compared to GTX line of thesame architecture. ECC ram & DP performanc is another strength - neither matters for Octane users, so why to pay more for features You're not going to use?

As for Your case, again, what is Your budget? What is Your case & PSU. Pluss, future plans fr upgrade..

Technically if You don't need much vram & can live with let's say 3GB then grabing something like 780Ti gives You the most speen, & since it's not the latest & greates architecture You can find good deal to buy them =)

Other than that, airflow matters, so depending on Your case & spacing between cards choose coolers acordingly. Non reference design coolers might give You slightly better performance if You have enough airflow while reference design will work the best if Your cards are placed in close proximity.

Here is nice example of custom rig by Polder Animation based on 5x 780Ti - this might give You some ideas. In future articles, like "Best GPU for Octane render" I'll group main GPUs based on performance, value & other aspects like cooling, upgradabity (aftermarket cooling options), architecture differences, etc.

[kraken]

Thanks, It was wishful thinking. I can get a Tesla for about $300. Since I'll never know what its capabilities are, then I doesn't matter what the cost is since it will most likely disappoint. I'll just save my pennies and try to get another 690. That's all this board and ps will support.

That's a nice rig Polder has. I'm not a good enough animator to warrant that investment. Maybe in 5 years.

Cheers.

[ChrisVis]

Chrisvis, where are youuuuuu?

Hi abstractmedia,

I have the same Motherboard and CPU now, but 64GB of RAM.
I got another Card, one GTX780 Ti... and my older 3xGTX590 and 2xGTX580... all working together in one system and the turbobox.
3xGTX590 and 1x GTX580 are in the turbobox, 1xGTX780ti and 1xGTX580 are directly on the motherboard.

I have a very fresh install and Windows 7 recognizes every GPU correctly in the device manager... so it is 9 GPUS in total.
BUT: Cinema 4D hangs on startup (with C4Doctane plugin installed), when all 9 GPUs are activated! When I deactivate one of them in the device manager, Cinema4D starts normally and 8 GPUs are working.
When I deinstall the C4Doctane plugin, it even works when all 9 GPUs are enabled... but only one is used by normal Cinema4D of course.

So where is the limit? Windows or the NVIDIA driver?
Should I give Win 8.1 a try?

Anybody else with more than 8 GPUs??

Greetings,
ChrisVis

From here: http://render.otoy.com/forum/viewtopic. ... ed#p174887

-

Although this thread seems to be outdated now, I can also confirm the 8 GPU limit for one system is not true.
I have all 9 GPUs in my system (PC+Turbobox) working. Got some issues in the past, because one GTX590 seemed to have glitches with one GPU, so I turned it off. Some weeks ago I luckily found another used GTX 590 beast very cheap and replaced the other one and now everything is fine.
I don`t know, if it is the same with 9 physically One-GPU Cards.
My full specs are in my signature. Still on Win7.1 64bit. Asus P9X79-E WS board.

I didn`t know about the 12 GPUs limit set by Otoy yet. Hope they will let it down. Cloud Rendering is also no option for me with less than 100kb/s upload stream.

Maybe I will get another card soon, probably the GTX 980ti when it is out next month, so I can try if my system can manage 10 GPUs.

Cya,
ChrisVis

[electric_eric]

-[/quote]

Although this thread seems to be outdated now, I can also confirm the 8 GPU limit for one system is not true.
I have all 9 GPUs in my system (PC+Turbobox) working. Got some issues in the past, because one GTX590 seemed to have glitches with one GPU, so I turned it off. Some weeks ago I luckily found another used GTX 590 beast very cheap and replaced the other one and now everything is fine.
I don`t know, if it is the same with 9 physically One-GPU Cards.
My full specs are in my signature. Still on Win7.1 64bit. Asus P9X79-E WS board.

I didn`t know about the 12 GPUs limit set by Otoy yet. Hope they will let it down. Cloud Rendering is also no option for me with less than 100kb/s upload stream.

Maybe I will get another card soon, probably the GTX 980ti when it is out next month, so I can try if my system can manage 10 GPUs.

Cya,
ChrisVis[/quote]


Hey,

I currently have 8 x 980ti recognised in device manager however only 7 cards show up in Octane or Geforce Experience.

I have the same motherboard as you and I'm running windows 8.1 64bit.

Did you have to disable much in the bios to get these 9 cards working?

Any help on this would be much appreciated as I currently have a 980ti sat on the table doing nothing!!

Thanks in advance.