Page 1 of 1

Lots of Denoiser Crashes

PostPosted: Wed May 08, 2019 8:56 pm
by labmeta
Hi Jaunjo,

I'm getting tons of denoiser crashes at the moment - Comes with an error message that there is not enough memory (needs approx 400Mb).
When i look at my system monitor i have over 32GB out of 64GB system memory free, and only 6.8GB out 8GB video card memory is being used.

The frame size it's trying to denoise is a fairly modest 3000x1000 ish - Any ideas? How much headroom does the denoiser need beyond this?

Best,

Paul

Re: Lots of Denoiser Crashes

PostPosted: Wed May 08, 2019 9:04 pm
by juanjgon
Hi,

The denoising is done on the GPU, so perhaps you don't have enough free VRAM to execute it. If you have several GPUs, you could configure only one for denoising, and never the same GPU used for the monitors. You could also try to enable the out of core feature, to see if it leaves more free VRAM for the denoising.

Thanks,
-Juanjo

Re: Lots of Denoiser Crashes

PostPosted: Wed May 08, 2019 9:34 pm
by labmeta
Hi Jaunjo,

Thanks for the fast reply. This is quite curious - If i enable out of core textures for the render node (I'm running everything through deadline here), deadline gives me a caught exception and the frame gets failed. De-enabling allows the frame to render but it then fails at the denoiser stage. I wonder if this caught exception could this be related to my previous support request about deadline that we never got to the bottom of?

Good idea to reserve one card for denoising ( this machine has x2 2080's) but sadly this didn't help. Further to this I have another machine with the exact same specs (x2 2080's) and it's rendering the same scene/frames without problem.

Any other thoughts?

Thanks,

Paul

Re: Lots of Denoiser Crashes

PostPosted: Wed May 08, 2019 9:48 pm
by juanjgon
Hmm, it is weird that this problem only happens in one machine if you have another one with the same specs. Are you sure that you are denosing only with the GPU not used for display? Usually, this GPU has more free VRAM. Do both machines have the same drivers?

Thanks,
-Juanjo

Re: Lots of Denoiser Crashes

PostPosted: Thu May 09, 2019 8:33 am
by labmeta
Yes same drivers, same brand of RTX cards too. I've just tried a driver update across all machines but it hasn't changed anything.

There's no other software running on either of the machines while rendering either.

The out of core switch crashing deadline nodes seems repeatable here, I'm reporting this to thinkbox as a bug to see what they make of it - I'm curious if it's repeatable for yourself or anyone else?

Thanks for you help so far,

Paul

Re: Lots of Denoiser Crashes

PostPosted: Thu May 09, 2019 8:34 am
by labmeta
and to answer your other question - I've tried reserving both card 1 and card 2 for denoising separately just to make sure - same error crops up.

Re: Lots of Denoiser Crashes

PostPosted: Thu May 09, 2019 8:57 pm
by juanjgon
What a weird issue, I can't figure what could be the problem, to be honest. And do you have this problem with smaller scenes? Perhaps this scene is just in the limits to leave enough vram for the denoising process, and somehow this system some a bit less available free vram.

About your problem with Deadline, how are you enabling the out of core feature? Does deadline have an option for it?

Thanks,
-Juanjo

Re: Lots of Denoiser Crashes

PostPosted: Fri May 10, 2019 9:48 am
by labmeta
Hey Jaunjo,

Thanks for the input on the denoiser issue - I'll work around it for now, it is a strange one. When i have more time i'll look into it some more.

For the out of core memory issue - There is no deadline option tick box or anything . As far as i can see it picks it up from the octane preference file for that machine. I'm enabling it from within the Octane Options from the Octane shelf in Houdini then closing the program. I'll test some more across the other nodes and see whats up.

Hope we can figure this one out!

Thanks,

Paul

Re: Lots of Denoiser Crashes

PostPosted: Mon May 13, 2019 9:28 pm
by juanjgon
The plugin has a couple of hscript commands to set the OOC parameters and to get the OOC status, as you can see in the attached image. They can be used also from python, and probably they should be included in the rendering script (for example in the Deadline hython rendering script) to configure the feature at the render time.

In any case, as you say, the OOC could also be configured from the preferences file, so I'm not sure why Octane crashes when it is enabled ...

Thanks,
-Juanjo