Cards shut down randomly :(

[Aselert]

Hello,

Am I the only one to have this kind of issue?

My graphics cards are sometimes stop working without reason!

This problem still persist after lot of months and headhache to found the issue. But I still can't found it and it makes me mad!!!
I'm explaining: When I launch a render, I mean a long time render (for few hours by example) with my render machine (3 GTX 580), it begins with 3 GPU, then 2 GPU... Then 1 GPU!!!??
I tried to changes cards, drivers, reinstall BS, change cards from PCI Express location, tested with 2 cards, verifying overheat etc... But nothing changes

I do not know what to do... Any help would be appreciated before I go really mad!

Thank you very much!

[ROUBAL]

It also happens to me with two GTX 580 3GB in a Cubix box. There is a difference of temperature of 5-6 °C between the two cards, because one occludes the airfan of the second card (as almost always in computers as well) due to no space between double width cards.

If I don't enable the EVGA Precision software which increases the airfan speed, the temperature can reach 86 °C and one of the cards can fail, and the second one a moment after.

Install EVGA precision or an other software allowing airfan control, and see if it still happen.

Please describe your workstation (OS/Memory/Nvidia driver...) in the signature field. It can help people to help you in some cases.

[FrankPooleFloating]

And tell us about your case fan setup. How many fans? Where are they located? How many spaces are available in your case to add new fans?

[Aselert]

Tahnk you for the replies.

Well, in fact it doesn't look like a overheat issue. It could be, of course, but doesn't looks like.
I'm explaining: it's really random: the cards shut down after 1min like after 1h, when it begin to render and the shut down appears from 70 to 90°C. And in the average, the temp is between 80-85°C at full load/charge. So really under the overheat stop (97-105°C). Or I'm wrong?

My system is:

I7 2700K
32Gb RAM
MSI Big bang Marshal
3x GTX 580 3072MB (Gainward)
Win 7 64 pro

The cards take 2,5 slots each so from 8 pci-e slots, i'm using 3 slots for 3 cards. And they have space, fans, and big box to breathe.

What is strange to me is the fact that the card/soft/win/driver show me nothing wrong: the card just stop to work. I can heard it easily and control it with GPU-Z: from 90-99% of load it goes to 0%... :/ And the app continue to work, but just with 3, then 2, then 1 cards... So the render time is divided each time!

Where do you think the issue is coming?

Thank you very much!

[whersmy]

what psu are you running?

[glimpse]

I would try changing PSU =) sounds power might be the cause.

[Aselert]

Mmmh maybe it's a PSUs issue yes. I don't know. But I have 2x850W, which is enough for 3x GTX580.

So if it comes from PSUs, the issue is because they are "poorly" connected?

Because one is connected to the mobo and the first GPU and the second one to the 2 others GPUs only.
I've just shunt the second with a cable to have an automatic working. Maybe the system have a limit.
But it's very strange because it "looks working" during the workflow, yes, and it stops after a time during the rendering. Maybe when the "watt charge" is to high.

So, thank you again, anybody have a solution to properly connect 2 PSUs? It could solve my problem and my nightmares!

(excuse me for my bad english, I'm french

[glimpse]

So if it comes from PSUs, the issue is because they are "poorly" connected?

if they are poorly connected some GPU's wouldn't work at all =) because one of PSU would get out of sleep, so I assume there could be something wrong with one of the PSUs.

What I would do? just try connect few cards & on one, then another PSU - if the problem is with one of them You'll get to know immediately. Try to put them under load & You'll see.

My PSU is out too - it's more than enough, but if I put system under load it just pass & sends system to reboot (so it doesn't matter even if I have extra power to put another GPU inside.

If they work on their own (let's assume this way) simply try to connect GPU's in different sockets (on the PSU if You have modular units or use different wires). Sometimes some rails (if You have multiple rails) & circuits are out of life.. & when they get load start triggering security built in causing shutdowns.

[ROUBAL]

Be aware that best efficiency of Power Supply Units is when they deliver 50% of their max power.

Some (most/all ?) PSU require a feedback signal from the motherboard to operate, so if you use a second PSU to feed something else than the mother board, this second PSU may not work properly or even not start at all.

Some PSU also don't start if they are not loaded enough (under say 15% of load). Usually, this necessary consumption is given by the motherboard. If you use a powerful PSU to feed just one additional GPU, it may not be enough loaded to work properly.

[Aselert]

Thank you very much Roubal
In fact yes, it "could" be the reason. I'm saying this with prudence because I had some "fail joy" before, thinking it was the solution... :/
In fact my system, except this, is very stable. BUT yes It's true, my second PSU is absolutely not connected to the mobo...
And to be honest it always surprised me this "out of mobo control" part system... Second PSU is only connected to the 2 and 3 gpus. So it has on link with the mobo.
I have to test to be sure but it makes sense to me, considering the situation.

If its right, guys, you're welcome to drink Champagne in my agency!