OTOY Forums

Smicha, here you go....it's not pretty though.

You're crazy Cuys! That's impresive setup =)
(though I still preffer cards inside the case)

That floats my boat

Now a serious question: you have 4xZ+1X which work in octane. What do you think, guys, if this IO space for such config is seen same as 5 gpus or 9 gpus?

Notiusweb,

Is there any chance you could connect more cards (borrow them for test purposes) to your system to exceed the 10 gpu windows limit and see if all of them work under octane?

Notiusweb,

What are these cables I marked with red? Could you share a link to the devices you've used to connect Zs externally?

I think these cables are USB3.0 cable. I have one of them 50cm but can be up to 1m without issue.

And about ressources used, we can post our ressources setting to compare and compile some information about that.

Hello all!

A) CABLE

Here is the cable
http://www.amazon.com/RIF6-PCI-E-Adapte ... e+extender

One caveat, it might downgrading bandwidth signal in order to achieve its goal.


B) 10 GPU - I have successfully tested adding another older card, GTX 660 Ti. It works fine!
I actually wanted to try 11 GPU, as I have another PNY Titan Z, which has an issue with one core, but my PC could not boot up with 11 GPUs going. I got "BF" error code on motherboard at pre-BIOS whenever more than 10 cores. I actually had to remove my CMOS battery to get motherboard working again. Don't know if this is what is causing the pre-BIOS "BF". It is scary, the PC just tries to reboot itself over and over. Maybe 64GB of ram would fix, I don't know...So, I think hardware-wise, for some reason 10 GPU cores will be my system's max? I never even could get to BIOS in repeated attempts when I tried more than 10 cores...

But, same riser, same cables, GTX 660 Ti (1 core) worked. And I could get the PNY Z working if I disconnected another Z, so it is definitely feels like it is a limit of the motherboard.

Too bad the 660 Ti has only 3GB Ram...bummer...I am now thinking of a 780 with 6GB, or another X with 12 GB and 3,072 cores...but I may as well wait for the next mega single-GPU card with 12GB and 6,144 CUDA. This is all me assuming that 10 cores was the limit, of course. (The mobo has 7 PCIE lanes, it just so happens I am using dual core Z).

Anyway, fun stuff...

Thanks for info!

I am still curious if there is any way to get 12 gpus (octane limit) working on a single mobo (amfeltec splitters, any splitters, raisers,....)

Thanks for all these infos. So I can hope to add a Z in place of the 580GTX.
here is what I gather on the I/O space :

titan : E000-E07F (IRQ 20) (03B0-03BB / 03C0-03DF) --> display
titan : DF80-DFFF (IRQ 28)
titan : CF80-CFFF (IRQ 2A)
580GTX : BF80-BFFF (IRQ 10)

Theses boards are on amfeltec:
titan black : 5F80-5FFF (IRQ 1C)
780Ti : 6F80-6FFF (IRQ 1E)
titan black : 7F80-7FFF (IRQ 1C)
780Ti : 8F80-8FFF (IRQ 1C)

Itou31, wanted to give you congrats for getting your set- up working! Great job!!!

Simcha, I would be interested to see a maxed-out GPU set up. If I could figure out my set up I could meet the 12 with my current Z#5 and 660Ti (9+3=12). If that worked, I could try another dual core card (in place of the 660 Ti) for 13, without ever needing to use a splitter.

In that regard, if anyone PC technical could think of why Im getting a "BF" looping boot error when trying to hook up the 5th Z, but not getting it when I hook up the 660 Ti, I would love to hear ideas. And it is not any one Z that causes it, rather it seems to be the cumulative presence if 5, in addition to the Titan X. Thinking because it is a pre-Bios error, might be a hard mobo limit, maybe an issue allocating power or IO to more than 10 GPU cores, maybe a DRAM thing with me only having 32GB vs 64GB, or a BIOS setting that if changed would allow it. Any ideas float'em by...Peace!

itou31 wrote:Thanks for all these infos. So I can hope to add a Z in place of the 580GTX.
here is what I gather on the I/O space :

titan : E000-E07F (IRQ 20) (03B0-03BB / 03C0-03DF) --> display
titan : DF80-DFFF (IRQ 28)
titan : CF80-CFFF (IRQ 2A)
580GTX : BF80-BFFF (IRQ 10)

Theses boards are on amfeltec:
titan black : 5F80-5FFF (IRQ 1C)
780Ti : 6F80-6FFF (IRQ 1E)
titan black : 7F80-7FFF (IRQ 1C)
780Ti : 8F80-8FFF (IRQ 1C)

I think you just might get that additional Z running.

Here's a little peak into my warped mind:
Try to find a pattern in the math that I’ve done with your system’s I/O space measures that you reported. If you see a pattern, but it’s not 100%, what breaks the uniformity? What video card function uses non-contiguous I/O space? Keep in mind that not only do GPUs use IO space, but so does other system hardware. So, to shed light into darkness, keep that in mind and you'll see the importance of looking at the resource space being used by your system's other functions when the road ahead otherwise looks blocked. That'll give you educated clues about what you might try disabling.

Guides:
1) Hex Counting to 15 = 1, 2, 3, 4, 5, 6, 7, 8, 9, A, B, C, D, E, F, …
Decimal Counting to 15 = 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15 …

2) Hex to Decimal Value [ http://www.binaryhexconverter.com/hex-t ... -converter ]

Pattern Hunting (I've re-ordered your listing - Why did I do it? How did I re-order it?):

Titan Black no. 1
5F80 = 24,448;
5FFF = 24,575;
24,575 - 24,448 = 127 span
————————————————
28,544 - 24,448 = 4,096 span
————————————————
780 Ti no. 1
6F80 = 28,544;
6FFF = 28,671;
28,671 - 28,544 = 127 span
————————————————
32640 - 28544 = 4096 span
————————————————
Titan Black no. 2
7F80 = 32,640;
7FFF = 32,767;
32767 - 32,640 = 127 span
————————————————
36,736 - 32,640 = 4,096 span
————————————————
780 Ti no. 2
8F80 = 36736;
8FFF = 36863;
36,863 - 36,736 = 127 span
————————————————
49,024 - 36,736 = 12,288 span ; 12,288 / 4,096 = 3; So I’d be looking to see whether this space is being used to any extent by any other hardware; if it isn’t, then the chances are good that you might install some additional hardware to use this excess, unused IO space. But how many additional GPUs might you be able to install in this I/O space gap? Three? Is this space being used by any other hardware? Or even if this space gap isn’t being used by any other hardware, is there another possible impediment to plugging in a third additional GPU processor (a properly functioning Titan Z has two GPU processors) . Sometimes, it pays to look ahead for a possible roadblock, particularly if you’re Hex counting. I’d quess two more at best, even though I got a quotient of 3. Why might I B right if you don't remove the GTX580?
————————————————
580GTX
BF80 = 49,024;
BFFF = 49,151;
49,151 - 49,024 = 127 span
————————————————
53,120 - 49,024 = 4,096 span
————————————————
Titan no. 3
CF80 = 53120;
CFFF = 53247;
53,247 - 53,120 = 127 span
————————————————
57,216 - 53,120 = 4,096 span
————————————————
Titan no. 2
DF80 = 57216;
DFFF = 57343;
57,343 - 57,216 = 127 span
————————————————
57344 - 57343 = 1 span
————————————————
Titan no. 1 (video display)
A) E000 = 57,344;
E07F = 57,471;
57,471 - 57,344 = 127 span
( I thought long and hard about re-ordering B and C, below, but settled upon leaving them together with A because it illustrates on the one hand an obstacle and, at the same time, a use option/function that you might be able to take advantage of in the future when planning on squeezing that last GPU processor into full compute functionality where you might have otherwise reached a roadblock - just change which GPU being used for display purposes.)
B) 03B0 = 944;
03BB = 955;
955 - 944 = 11 span
———————————————
960 - 955 = 5 span
———————————————
C) 03C0 = 960;
03DF = 991;
991 - 960 = 31 span

To my warped mind, all of those "4,096"s remind me to always activate in bios any 4g/high memory functionality.


What motherboard are you using?