rendering error " CUDA error 700 .." Tweet

[Ebo]

Hi, I keep getting this error after I up a render resolution to 4096x1716 and max samples to 2048. I still get to render first 3 sometimes 5 frames but then it stops with this error. Anything I can do to stop that? I have one GTX1080ti. Thank you. Jiri
------------------------------------------------------------------------------------------------------------------------------------------
*** OCTANE API MSG: CUDA error 700 on device 0: an illegal memory access was encountered
*** OCTANE API MSG: -> failed to download symbol(stats data)
*** OCTANE API MSG: device 0: path tracing kernel failed
------------------------------------------------------------------------------------------------------------------------------------------

[juanjgon]

Hi,

Sorry, it is hard to say what could be the origin of these 700 CUDA errors. Usually, they are related to hardware or drivers issues. Disabling any overclocking in the system or improving the system cooling can help, but the problem can be related to a broken GPUs or problems with the PSU. You could also try to update or downgrade the NVIDIA drivers to check if another version improves the system stability.

In any case, if you can reproduce always the same problem and you can share the scene with us, I could take a look at it here. What Octane version are you using?

Thanks,
-Juanjo

[mlmcgoogan]

I've also just started running into this issue. What's the recommended Nvidia driver version for the current 2019 release?

[mlmcgoogan]

Something I've noticed with further testing - My system memory (not GPU) seems to keep growing. It looks like a chunk gets allocated with each new frame, then part of that chunk is deallocated moving to the next frame but not all of it, so my RAM usage just slowly grows frame by frame until I'm at about 100% system memory usage and then the render will just sort of get stuck rendering mid-frame, it will just sit like this seemingly forever. When I "Interrupt" the render is when the error occurs.

The way the memory usage grows looks like a memory leak but I'm not sure what's happening behind the scenes.

[mlmcgoogan]

Sorry for the repeated posts, just gathering more info from testing.

Plugin version: 2019.1.0.5 AND 2019.1.1.0
Octane API version: 2019.1
Houdini HDK version: 17.5.360
NVidia driver: 431.36

The problem appears to be associated with the "Rendering mode" set in the Octane ROP. If set to "Update scene" the problem does not occur, memory usage is stable. If set to "Full scene reload" then the memory usage will grow each frame until freeze. Note that I've had it set to Full scene reload for this current scene because one of the geo nodes doesn't seem to update in IPR or MPlay frame to frame without a full reload (It's currently just displaying poly geo loaded from a file cache sop, not sure why it doesn't update without a full reload.

Some additional info - I have 64GB system memory. When I start rendering Houdini might be using ~4GB. When the render kicks off it will then start growing - note Houdini's usage is growing, not MPlay. If I interrupt the render at around ~48GB usage, that 48GB will stay there, I can interact with the scene and make it shift around by hiding/showing objects, but it all remains allocated. If I play through the frames in Houdini (not rendering), the memory usage does not grow.

When I resume rendering, the memory usage will then resume growing, frame by frame. Also note that when I start the render back up (via "Render to Disk") there's a moment where the RAM usage for Houdini drops down to single digit GB but then pops back up to wherever it was before (~40s GB in this case).

Also note that out of core textures is disabled but if I enable it, the problem is still there.

[mlmcgoogan]

Last update. Turned on Octane console logging.

The reason that one geo node isn't updating with a simple scene update, but requires a full scene reload:

"The object "geo1" can not be updated, it has a variable mesh structure along the frames"

Not sure if that might be related to the memory leak or not (or what that actually means/how to resolve).

[juanjgon]

Hi,

This error means that currently the Octane plugin can't update a mesh object because it doesn't have a constant topology along the sequence. To render these kinds of objects you need to configure the rendering to full scene reload. I'm not sure if this could be related to the memory leak. I'll investigate it, but if you could share your scene, it could help a lot.

Thanks,
-Juanjo

[mlmcgoogan]

Hey juanjgon,

I'm not able to share that scene but I am able to recreate the apparent memory leak in a simpler scene. In fact it looks like it has nothing to do with the variable mesh structure issue, it may be that there's a memory leak in any scene with a full scene reload when you render a frame sequence out do disk (using MPlay).

Attached is a simple scene where the RAM usage will grow every frame. Note that it's not leaking the whole context frame to frame so the allocated memory does drop moving to the next frame, but there's a net growth each time. It's significant enough that you don't need a particularly complex/heavy scene to run out of RAM in 100-200 frames or in the case of my heavier scene after about 35 frames.

Had to share via my Dropbox, for some reason whenever I try to upload an attachment the page just resets after I complete the CAPTCHA
https://www.dropbox.com/transfer/cap_pi ... xclTuwhDys

Some notes on my system:
Plugin version: 2019.1.1.0
Octane API version: 2019.1
Houdini HDK version: 17.5.360
NVidia driver: 431.36

[mlmcgoogan]

Hey juanjgon,

Just wanted to check whether you were able to grab that scene file before it expired or if I need to re-upload? Thanks! Again really appreciate all the work you put in for these Houdini plugins!

[juanjgon]

Don't worry. Thanks. I can reproduce the memory leak while rendering scenes in full scene reload mode without problems, but I'm still working on the fix.

Thanks,
-Juanjo