OTOY Forums

itou31 wrote:

Notiusweb wrote:
Do you have the fabled 12 GPU now (or at least the ability to have)?

No way, I can only put 9 GPUs. when try the 10th, cannot boot (BIOS error 0xd4 : out of ressources) and no 4G option on my P9X79 deluxe
Ha also, I mod another Titan single slot to put in my main PC.
So now 4 internal GPU and 5 external GPU.

I am trying for 14 but getting the d4 error pre-Bios, no matter how I arrange my settings and gear as well. I am going to investigate with ASRock, my mobo's manufacturer

I am still waiting to hear back from ASRock. They had one last BIOS update for my board that did something new with adding compatibilities for SSD on the PCI lanes, but it doesn't work with Titan X.
They are quite bureaucratic about handling issues, meaning it has to be very super specific when you communicate with them. I asked for help adding a 14th GPU and asked on the side if they could explore releasing a version of the newest BIOS to work with Titan X. But I am going to play along with the bureaucracy and send in a separate ticket requesting fix for using the newest BIOS with Titan X, and follow that progress course alone.

Itou, have you been using the Tonemapping feature in devices? It does work to stop crashing with baking cam and when Post Processing is on, BUT you have to at least have one card that is able to render V3 baking cam and post processing without crashing, and you set this as the GPU for tonemapping. So, I can have my 6 GPU on Amfeltec and 6 on USB riser like before, and just have tonemapping be handled by (at least 1 of) the USB riser cards. But like, if I had say 2 Amfeltec clusters with all 12 GPU on them, I would then have to always have my Titan X be a part of the rendering as the tonemapping GPU, since Amfeltec cards all crash. But since my USB riser cards did not make the V3 crash, I can use them and have Titan X sit out as the display card like I usually do. I still haven't brought the Amfeltec back into the mix yet because the plugins are not yet up top date with the selectable tonemapping option (although, Daz Studio plugin does already have!). But once they have, I can go back to using the Amfeltec in any application and not have to worry about V3 crashing when baking cam or post processing is used.

Hi Notiusweb,

As I post it earlier, since I have changed my amfeltec splitter to the cluster, there is no crash with V3, so if something like this happen again, I will check Tonemapping to the one that not crashed. What is the performance lost if you only check one GPU for tonemapping ? This process is only happend to the end of the render ?

Hi itou,
There is no performance lost, or gained for that matter, when you have just 1, vs 2, vs 6 GPU Used as tonemapping.
So it's almost like a ceremonial thing.

As far as end of render, you can just put it on as your default Config, and just use that Config indefinitely for all projects.
Ie, tonemapping on the 6 GPU from the riser turned on, and tonemapping on the6 GPU for the Amfeltec turned off...
(But, interestingly enough, an earlier v3 crash workaround I used to have for single frame renders was to turn off post processing and then when the render finished using all 12 GPU, I would uncheck all Amfeltec cards and then turn post processing on for the finishing touch.).

Hi guys.
Help me, please.
What motherboard I need for 8 external GPU and 2 internal? I use amfeltec clusters.

Thanks.

ertaro wrote:Hi guys.
Help me, please.
What motherboard I need for 8 external GPU and 2 internal? I use amfeltec clusters.

Thanks.

Sorry for delay. Here's what Amfeltec told me:

"The motherboard limitation is for all general purpose motherboards. Some vendors like ASUS supports maximum 7 GPUs, some can support 8. All GPUs requesting IO space in the limited low 640K RAM. The motherboard BIOS allocated IO space first for the on motherboard peripheral and then the space that left can be allocated for GPUs. To be able support 7-8 GPUs on the general purpose motherboard sometimes requested disable extra peripherals to free up more IO space for GPUs. The server type motherboards like Super Micro (for example X9DRX+-F) can support 12-13 GPUs in dual CPU configuration. It is possible because Super Micro use on motherboard peripheral that doesn’t request IO space."

I have three Supermicro X9DRX+-Fs [ https://www.supermicro.com/products/mot ... drx_-f.cfm & http://www.superbiiz.com/detail.php?name=MB-X9DRXFB ] based systems. The second to last sentence in the above quote (that contains "12-13 GPUs") applies to an X9DRX+-F system running Windows. Linus OSes, like Linux Mint, support even more GPUs with the X9DRX (or X10DRX). Whether the GPUs are internally or externally placed doesn't really significantly affect the IO space limitation, except that it is possible to place more GPUs internally in a system with a LIan LI D8000 case with an X9DRX motherboard.

ertaro wrote:Hi guys.
Help me, please.
What motherboard I need for 8 external GPU and 2 internal? I use amfeltec clusters.

Thanks.

I think mathematically it would need at minimum 4 PCIE slots (2 for 2 separate 4-way Amfeltecs), and 2 for the internal cards. Although with just 4 slots, spacing may be the issue, so you may choose to go higher.

Hmm, you don't know, could Asus X99-E WS run with 8 gpu cards?

ertaro wrote:Hmm, you don't know, could Asus X99-E WS run with 8 gpu cards?

It has 7 PCI slots, so with Amfeltec, at minimum physically connecting 8 is possible.
If you listed here the current configuration you are running with 6 GPU (ie how your 6 GPU are currently connected to the motherboard)), then we can see what it is you have going on.
First step would be for you to detail what your rig is currently.

Ertaro told me he can't get it to work with 7 GPU, but he has an ASUS board (X99 e WS). I told him he might need to ask about bios settings to enable 4G decoding but I'm not sure how it worked on an Asus board.
Think he might need that if he asks here again.

Edit: He since told me he only gets up to 6 GPU, even with 4G enabled.
In case he comes knock knock knocking on Heaven's doh-ooh-woor, hey hey, hey-hey-yeah...