External Graphics Cards PC Tweet

[A Polish Ginger]

Smicha
Thanks for the recommendations, I didn't know about PLX chips. Though I am going to try out the P9 X79 E-WS, which does have 2 PLX chips only because I don't feel like buying a new CPU if I can help it. Also the GPU Cluster ships tomorrow, but I don't know how fast it will arrive. Fingers are crossed that I'll get it this weekend, but most likely will be next week. Canada to the States.

Notiusweb
Thanks and I was originally planning on using the 1st PCIE lane for the display card. Because I have 6 PCIE lanes and the separation between lanes 1 and 2 are big enough to fit a dual slot card, though that won't be the case with the new board. I currently have the P9 X79 WS so I'm getting the step up from that and new so hopefully I can at least get all 6 GPUs up and running with little trouble. The new board should come this weekend so I'll update once I'm able.

I will attempt your suggestion, but on the current board I was only able to get those 5 GPUs to work once. I tried it again this last weekend and it didn't decide to want to work with me. So that is what lead me to do that single riser test. My thinking is since each lane is not working with a single riser/ single GPU connected to it (except lane 1) Then I can think of 2 problems.

1. Since the riser works with lane 1, but can't with the others. The amount of bandwidth is to low to lanes 2-6 and can not function properly with the 1x USB 3.0 riser.

2. the amount of bandwidth in the riser needs to be bigger 4x, 8x, 16x in order for the lanes to accept it.

These are complete guesses, so I could be completely wrong. Is there a way to manually set bandwidth on the PCIE lanes? I know that the CPU decides bandwidth automatically, but I was thinking if you set it to x1,x4,x8,x16 manually it would take out the guess work for the CPU and make things more fluid. Though I'm just making guesses here. What really bothers me is that 4 GPUs directly connected doesn't working anymore, because last time I got 4 980 ti's to work no problem. I just plugged them in and that was it. Which also leads me to believe the bandwidth priority of this Motherboard is screwed up.

I'll update once the new board arrives and I have the time to install it to see if I have the same issues, hopefully I don't.

Also as to you waiting on the response from your BIOS manufacture do you mean the Motherboard manufacture or is there a different company that makes BIOS? When I went to contact ASUS about maximum GPU load they kept on saying 4 only. No matter how I phrased the question they kept telling me 4 only on all their Motherboards. It was over chat support so I don't know how to hold his credibility/ knowledge on the subject. I might of gotten a bad person maybe, but since what we are doing is breaking/ bending rules I don't know that your average technical support will know the details and just read what is off the box. Perhaps if I explained the situation to the person better we would've gotten somewhere, but that is something to try a later day. You'll just have to keep calling until you get someone that knows what you are talking about. This is assuming that you're talking about the Motherboard manufacture. If you are not then those people most likely are well versed... hopefully.

[smicha]

Manual pcie speed setup - A Polish Ginger is asking - has been on my mind for some time.

My further questions about pcie speed: I read in asus manual that it sets x16 x8 x8 pcie speeds for 3 gpu layout. How Octane 'reacts' on it? If the fastest card (with x16 assigned to it) loads geometry twice as fast as the other two gpus (x8) will the x16 gpu 'wait' for geometry upload to the slowest gpus? Shortly speaking: shouldn't the pcie speed be the same across all gpus? And if so, if the minimum pcie speed is x4 (to handle a gpu using a riser) then it is not possible to run more than 10 gpus on a 40 pcie lane cpu, e.g. 5930k, 5960x. To do so, we'd need 2x xeon mobo...

What do you think guys?

[itou31]

I don't know about how speed is managed. I have a 3930k and an asus X79 deluxe. Good to know if we restrict PCIe lane on each GPU installed, then we can install more GPU !?
My external power supply 1600W has finally failed to power 2 titan black and 3 780Ti. I just know by testing with OR standalone with the free corner from Johannes. The power supply just shut down itself after 10 seconds of rendering. It never failed before. For stability test I use octanebench. And I use DAZ with octane plugin. Also with max, it's ok.
Just test with one 780Ti disable and it's OK. I need to re-think about power distribution.

[Notiusweb]

Hello all! Here is my latest:

Up to this point whenever I try to boot with > 10 GPU I got "bF" error, i.e. using a Titan X as primary GPU and 5 external GPU Titan Z (11 total GPU CUDA devices).
My BIOS manufacturer ASRock suggested using an older revision BIOS in order to enable an "Above 4G Decoding" (ROM option allowing usage of 64 bit devices, see https://en.wikipedia.org/wiki/PCI_hole, and, https://en.wikipedia.org/wiki/Memory-mapped_I/O). Most recently I could not try because the older revision BIOS they were suggesting did not support the GTX Titan X. So, I subsequently tried using the older revision BIOS with my GTX 660 Ti, and I have confirmed that under the suggested older revision BIOS, I can, yes, get PC to boot with 5 Titan Z. There is an option under NorthBridge Configuration "Above 4G Decoding", by default is "Disabled" and when set to "Enabled" it booted past post and into Windows with 5 Titan Z! Upon doing so I hit what would appear to be an OS snag of 10 GPU, I am thinking it is a Win 7 64 I/O thing, but I didn't bother to sort through that, I just wanted to see if this whole thing is even possible. In my case, I am interested in having the Titan X (12Gb) in the picture as the primary GPU because there is a noticeable improvement versus the 660 Ti (3Gb) in apps like SketchUp, Blender, and Daz Studio. Plus I used to get a lot of errors, such as OpenGL and timeouts, with the 660 Ti, that I no longer see using the Titan X.

Some details during my testing:
-newest revision BIOS P3.30M, with 4 Titan Z - 660 Ti / Titan X as primary boots successfully
-newest revision BIOS P3.30M, with 5 Titan Z - 660 Ti yields beeping error "d4", whereas Titan X yields "bF"
-older revision BIOS P3.30F, no 4G enabled, with 4 Titan Z - 660 Ti yields error "bF"
-older revision BIOS P3.30F, 4G enabled, with 4 or 5 Titan Z - 660 Ti boots successfully
-older revision BIOS P3.30F - Titan X as primary doesn't/can't boot, yields error "b2"

Conclusion: GTX Titan X cannot boot as primary GPU with older revision BIOS P3.30F. Newest revision BIOS P3.30M, allows Titan X to boot as primary, however has no option in either Boot Menu or NorthBridge for "Above 4G Decoding". As such, I cannot try 4G Option with a Titan X in the scenario. I am already looking to see now if ASRock can get the 4G option added to the newest revision BIOS, but I can now pass the data onto them regarding success with the 660 Ti.

I guess my testing won't be a true "Can I do it?" scenario, it will be more of a "Can I do it the way I want it?" scenario, as I am interested in the Titan X as the primary GPU.
In the meantime, here is a girl with a sword and an eagle floating mid air...

Regards!

[smicha]

I don't know about how speed is managed. I have a 3930k and an asus X79 deluxe. Good to know if we restrict PCIe lane on each GPU installed, then we can install more GPU !?
My external power supply 1600W has finally failed to power 2 titan black and 3 780Ti. I just know by testing with OR standalone with the free corner from Johannes. The power supply just shut down itself after 10 seconds of rendering. It never failed before. For stability test I use octanebench. And I use DAZ with octane plugin. Also with max, it's ok.
Just test with one 780Ti disable and it's OK. I need to re-think about power distribution.

A 1600W PSU shall be enough to power 5 gtxs (the system shall draw no more than 1300-1400W) . The shutdown may be due to power efficiency of your PSU. What is the model of the PSU?

[smicha]

My BIOS manufacturer ASRock suggested using an older revision BIOS in order to enable an "Above 4G Decoding" (ROM option allowing usage of 64 bit devices, see https://en.wikipedia.org/wiki/PCI_hole, and, https://en.wikipedia.org/wiki/Memory-mapped_I/O)

Thanks for sharing.

BTW. I'd get rid of 660 definitely...

[itou31]

A 1600W PSU shall be enough to power 5 gtxs (the system shall draw no more than 1300-1400W) . The shutdown may be due to power efficiency of your PSU. What is the model of the PSU?

I have the LEPA G1600. I think about the 12V sharing rail that has some trouble.

Edit : Yes ! Just check the 12V rails sharing and update the output to the last 780Ti. And OK, no shutdown now. Have all my 8 GPU working at max load.

[A Polish Ginger]

Gentlemen the new Motherboard and the GPU cluster unit and I can say that it ALMOST worked Out of the 6 GPUs I have I got 5 of them to work and the 6th one is recognized, but gets an Error code 43.

As for the Amfeltec GPU Cluster so far it has no problems. I'm questioning their instructions on relying on the backplate for solid structural integrity while mounting the bottom GPUs, but after that ordeal of me worrying that the backplate would snap it, it fit pretty snug. Once assembled the structure is solid, so no complaints here. As for the performance of the device it worked the first go. I had installed first 2 GPUs (I was going to do one at a time but you need to take off a arm in order to "safely" plug the GPU in, so I did 2 at a time). Installed drivers and worked fine. Then I added 2 more and was totally functional. Then I tried to add my 6th card.... that is when I hit a wall.

I was able to boot into windows and when I went to check the devices in Device Manager I got that little yellow triangle with a Error Code 43. So it "sees" it (Recognized as a 980 ti) but dome driver is not reaching through or something like that. Thus began the 5 hour troubleshooting session to lead to no avail. I tried...

1-All accepted BIOS versions with 4G Enabled
2-Updating/Installing/UnInstalling Display Drivers
3-Different PCI-E Slots with the 6th GPU
4-Tried Gen 2 PCI

1- As for the BIOS related issues only one version would boot windows. All other versions would go into a boot loop and would stop in this sequence d4 97 d7 on the DeBug LED. Through a quick Google search to just find out I could look in my manuel for the codes meanings, They are
d4- PCI resource allocation error
97- Console Output devices connected
d7- No Console input devices found
Then it would repeat after stopping at each error code for a couple of seconds.

2- I did this through the Device Manager and Display Driver Uninstaller (DDU) I spent a solid hour disabling, update drivers, sending it back to a VGA Video Adapter to a 980 ti, launched in safe mode with DDU then reinstalled, etc. The one thing that really bugs me is that I thought, during installing the drivers one of the times, that the triangle of my rage disappeared but it appeared again after the screen stopped flashing. Also during this test I was able to get windows to try and update the card, but it failed. With a bunch of Drivers that I did not install with the motherboard (I did try installing all the motherboard chipset drivers to see if that worked, but it didn't). This leads me to believe that windows is halting the GPU from operating.
I did try to look for how to registry hack, but it was brief and the only vid I found was how to do it for a USB device. So if anyone knows how to registry hack GPUs send that info this way, because I have a strong feeling that is the solution to this issue.

3- Self explanatory, moved the GPU around still was not at operating properly. Though now that I think about it I probably should of moved around the other GPU and PCI host card.... Eh I'll try it later.

4- The only options I found relating to PCI slots (at least to my eyes, basically anything that said PCI in it I started to mess with) was 4G Decoding, PCI Gen Selector, Intel Visualization, and PCI Spectrum. I didn't really touch PCI Spectrum cause it said that it was for OC-ing and was already set on Auto so I didn't mess with it. The Intel thing I tried a couple of time but didn't do anything noticeable to me. Always had 4G Enabled. Then I tried to set all the PCI lanes back to GEN 2 and when I reset my computer it would get to the screen loading Windows then go black and start the boot process all over, though the error box appeared where it showed me if I would like to start windows normally or Repair. If I did start normally it would loop and I didn't press repair. I might try that, but I don't know if it would really screw me over or not so I decided against it for now.

Conclusion- I feel the solution lies in the registry hack that I have heard of, but I do not know how to perform so if anyone knows/ knows a link that would be beautiful. I also did read up on how bitcoin miners where using "dummy monitors" have no idea what that is, but sounds intriguing. I still kind of find it funny that the GPU Cluster works but the GPU directly plugged into the board is not working. I just did a fresh install of windows too... Maybe I should try without installing updates see how that works, or at least the majority.

Anyways still happy because it at least see it and can hold five GPUs solid so far. The GPUs I currently I have working are 1 Titan and 4 980 ti's (5th 980 ti is the bother, scored 550) and (as you can see in the pics) I have the Titan in PCI lane 1 and the Host board in Lane 3 (Lane 2 covered up by Titan).
But I'm thinking its going to be like how Notiusweb said "Can I do it the way I want it?" scenario. I wish I just understood the problem better and where I can go/ what to do to fix it.

Also don't PLX chips expand on the CPUs PCI Expansion?

[Notiusweb]

Hello!

My update:
I sent request form to ASRock for a TitanX-compatible-Above-4G-Decoding option, and received a new BIOS same day, P3.30K. I tested and it did not work with the Titan X, I received the same "b2" error that I get with all the other non-Titan X compatible BIOSes. The "b2" error merely comes up because it is the TitanX, which is separate from the "bF", which comes up when >10 GPU are boot up without a 4G Decoding enabled.
I have to admit, I was suspicious given that the only TitanX-compatible BIOS is the latest P3.30M (with 'M' coming after 'K' in the alphabet )
However, everything works with my 660Ti, so I tried enabling the Above 4G and then again installing the TitanX, but still "b2" error. Having the 660Ti is pretty valuable as a reference since I have it to bail me out of non-working BIOSes that I otherwise could not boot out of.
I certainly appreciate ASRock following up with me so quickly. I have already relayed back that the new P3.30K is non-compatible with Titan X, and that I welcome further testing. I will see what they say...

Polish Ginger:
I think that is encouraging that you have a viable set-up now utilizing the Amfeltec cluster. The 5th 980Ti GPU is being recognized, even if not enabled yet, so it is just a matter of getting the last one sorted out. I do not know of a registry mod that enables a non-working display adapter in device manager (with the triangle), I only know that there is a registry mod that can let Octane Render itself recognize an enabled display adapter in device manager if Octane was not already recognizing it as an available CUDA device in the preferences menu of Octane.

I would try 2 things:
(1) set aside the lone 5th 980Ti GPU (the '6th' total GPU) and get a pulse on how well everything works with using the primary Titan and the 4 in the cluster. In other words, establish that the cluster is working consistently. Then,
(2) set aside the cluster and see if this last 980Ti in a PCIE lane by itself works as the only external GPU with the the primary Titan.

If you see that the 4 in the cluster and the lone last 980Ti work independent of each other, but not together, it may be a non-GPU specific issue involving > 5 total GPU (1 primary, 4 external). Possibly you might in this case find that it works to have 1 primary GPU Titan, 3 on the cluster, and 1 on the PCIE lane. or 2 in the cluster, and 2 in lanes, etc...in other words, any combination setup that does not exceed 5 would work. In this scenario, any registry mod targeting a single GPU instance may not make any sense.

BUT, if you see that this last 980 Ti does not work by itself, that could be an easier situation to try and tackle (given it is not broken), because it may just be underpowered somehow, it may be something specific to just this card being recognized by the OS, or it may be something to teak with the setup in BIOS. If this is the case, get it sorted out separate from the cluster's 4 being active.

It would be a starting point to identify that it is either a 'system thing' vs a 'card thing'...

(NOTE: I actually put stickers/tape on the cables and cards with numbers just not to get confused sometimes when working with multiple versions of the same card! )

Anyway, your progress is ahead of mine, I am still in BIOS world, you are in OS world...congratulations! I am very interested to see what happens with your rig as I know cards not being enabled in the OS is the next puzzle I will have to solve. It appears that itou31 has seen that OS itself can be the barrier sometimes.

[A Polish Ginger]

Gentlemen I have to say that I'm just dumb..... I didn't have the 5th 980 ti plugged into the PSU unit................ it was so simple that I started to laugh

But yeah Phase 1 of the DTRM (Dream Team Render Machine) is in operation

Anyway, your progress is ahead of mine...

HA, you got good jokes. I was here struggling with a mere 6 GPUs when you have greater than 10. I'm no where near your level so don't sell your self short. I hope that some day I'll have around that number but keep up the updates.

Not just Notiuweb, but anyone that is going through this struggle. I've learned so much just browsing this forum, that I wish to thank you all from my non existent soul (haha get it). But seriously keep it up gentlemen!

Sincerely,
A Polish Ginger