Pascal CUDA Problems and troubleshooting

[mikeadamwood]

OK so you may of read my other posts regarding this so thought I'd post this information to any others having problems with the CUDA errors on pascal cards. Please don't think I'm an expert in hardware, because I'm not but I have learnt a lot in the past few weeks. I just think if this post can help anyone then it might as well be here to avoid going through what I've been through.

So since getting my duel Evga Hybrid 1080(Pascal) rig I've been bogged down by CUDA errors..But I've finally got it stable and working after a few weeks of pain. I couldn't work like this as I was getting them every 5 minutes, and rendering an animation would end up in a freeze so it was totally unproductive.

So the error is specifically...
device 0: detected an error on render device! trying to recover...
CUDA error 700 on device 0: an illegal memory access was encountered
-> failed to deallocate device memory

I have had a few others but this is most common.

HOW I TROUBLE SHOOT

First step was to update the BIOS after advice from Smicha, I'm not sure if this helped in the long run but it may of contributed. Now this may depend on manufacturer but mine is an ASUS X99-e so I updated to latest BIOS version which was 3402 at time of writing.

You can find this information by going to system information in Windows then cross compare with manufacturer driver/BIOS versions to see which one your on.
In my case (https://www.asus.com/uk/Motherboards/X9 ... _Download/) Pick OS > Drivers > BIOS

I then cleared all my Nvidia drivers AND Octane plugin and Octane Standalone then proceeded to shut down my machine. I then removed ALL my cards, just to note I have a 980ti for display and 2x 1080 Hybrids. I then put the 980ti in and booted up machine to see if this card worked....Installed necessary drivers and also installed Octane Standalone Version 3.05.1 and Octane C4D 3.05.1

I rendered and everything was fine, but my intention was not to use 980ti as render card so no extensive test was made. I then shutdown and put one of the 1080's in to test it alone. I tried with the latest Nvidia drivers but got a crash/CUDA error with one of the 1080's ALONE...I was only doing 20 minute benchmark renders so this couldn't really be enough evidence to prove much...But I was definitely getting more than I had been. So 20 minute on each card, one failed and one didn't which was strange.

I made sure to label each card with a sticker while I was doing these tests...

So once GPU's tested to see if there was a hardware issue. I determined that one of the GPU's could be faulty as every time this 1080 was in it would error, but other one wouldn't. So that is going back to manufacturer and a 1070 is in its place for the time being till replacement.

I spoke to Steven Emerson(YouTube) who has a quad 1080 rig and same mobo and he said he'd been lazy so hadn't updated his Nvidia driver since version 373.06 so I decided to match that and install that one, to try and match his setup as close as possible, as well as Octane version 3.05.1 like I stated previously.

So now everything is rendering fine and haven't had an error since. I've rendered 300 frames of an animation so far and haven't once had an error in live viewer.

Like I said, I'm no expert so some of these steps may be completely irrelevant to the problem, but I thought it was best to document the steps I took. It could of been a faulty card, but could also be the driver or even the BIOS update or mixture of any. I may even be able to update my drivers and it still work now, but I'm staying put while it's all in working order.

So to recap:

Hardware
If duel GPU setup, test each card on its own to see if there is a faulty card as this will bring down your whole render.

Software
- Update BIOS
- Roll back NVidia Drivers to driver version 373.06 for pascal
- Roll back Octane to version (Octane Standalone Version 3.05.1 and C4D Octane Version 3.05.1 works for me)

Big thanks to Steven Emerson and Smicha who really helped me alot with the hardware stuff!

[bertritude]

I don't know if it will help you, but on a SLI of GTX1080, I had every time on render a cuda 700 error.

the problem was I overclocked my cards a bit too much and it made octane crashing.

I underclocked a bit and now it works perfectly.

[mikeadamwood]

I don't know if it will help you, but on a SLI of GTX1080, I had every time on render a cuda 700 error.

the problem was I overclocked my cards a bit too much and it made octane crashing.

I underclocked a bit and now it works perfectly.

Thanks bertritude but the cards are factory set, no overclock. SLI bridge is also non existent as I don't really game.