VRAM Not Adding Up

[AaronWestwood]

Hey all,

Been noticing this more and more as of late across all my projects. I have 11GB available, but as you can see in the screenshot below it's coming in FAR below this.

Latest studio drivers (with a full DDU clear prior), windows drivers, Octane version. No apps are open in the BG with a fresh system restart.

I notice when I open the project and start working I have access to all 11gb (8ish after windows), but the longer I work/toggle layers/etc that FREE amount gets lower and lower, eventually leading to a CUDA crash.

I do environments, so all that's in my scene is a height map displaced, scatters and a HDRI. Nothing too crazy, and going back a month or so I had never encountered this.

Any ideas on this?

Thanks for your time!

AW

[AaronWestwood]

As a follow up, restarting C4D and not changing the scene at all, the attached screenshot now shows the correct values.

Again, I changed nothing other than restarting. However as I said above this number will go down as I work leading to a crash (without adding anything to the scene)

Weird?

[AaronWestwood]

Final post with a crash log. This happened with only 4.2gb used.

[glimpse]

Good day Aaron,

what that error say is that Your mesh does not fit into GPU memory and RTX starting to have problems.

one way to solve this would be to actually disable RTX (no RTX boost anymore), so even if mesh or textures goes out of core, You would still be able to render with a bit of penalty.

if You see huge RTX boost, from RTX On, might be wise to do disable couple of cards, set NVLINK between other two and finish rendering with NVLINK (& RTX enabled) on two cards rather than Out of Core (without RTX).

not sure if that makes sense, but basically Your scece is close to the edge and seems that RTX is freaking out - also might be the cause why You see VRAM being messed up (just a guess).

let us know how it goes.

Sinc.
tom

[AaronWestwood]

Good day Aaron,

what that error say is that Your mesh does not fit into GPU memory and RTX starting to have problems.

one way to solve this would be to actually disable RTX (no RTX boost anymore), so even if mesh or textures goes out of core, You would still be able to render with a bit of penalty.

if You see huge RTX boost, from RTX On, might be wise to do disable couple of cards, set NVLINK between other two and finish rendering with NVLINK (& RTX enabled) on two cards rather than Out of Core (without RTX).

not sure if that makes sense, but basically Your scece is close to the edge and seems that RTX is freaking out - also might be the cause why You see VRAM being messed up (just a guess).

let us know how it goes.

Sinc.
tom

Thanks alot for your reply Tom.

Rendering this scene with/without RTX is with 30 seconds, without 1min40seconds so RTX is needing to stay.

Wondering if you might have any further insight. With RTX ON, a fresh launch of Cinema and a fresh launch of the scene i'm working on. All assets are on and I have 5.15 used, and 3.5 free. So with all assets on I still have 3.5gb free (after the fresh launch).

How could it be that through working on the scene (adjusting daylight/material balancing etc) I then am running out of memory and have to relaunch cinema to get my 3.5gb back?

AW

[frankmci]

How could it be that through working on the scene (adjusting daylight/material balancing etc) I then am running out of memory and have to relaunch cinema to get my 3.5gb back?
AW

This does not directly help your situation, but it sounds like you have what is called a memory leak, and understanding what that is might be useful. Essentially, as data gets moved around, written, read, deleted, sometimes the software loses track of little bits of free space, recording it as in use or otherwise unavailable, when it actually should be available. This often happens in very small pieces, but the longer you work, the more operations you do, and the more times memory locations get written and erased, thousands of chunks of data get shuffled around, those little small errors add up. Eventually, the amount of free memory drops below the threshold required for proper operation, and crashes, odd behaviors, etc. kick in.

There can be many layers of abstraction in the code managing memory, from firmware, to drivers, to OS, to application, and even multiple layers within those, so tracking these kinds of errors down can be especially tricky.

https://en.wikipedia.org/wiki/Memory_leak

[AaronWestwood]

How could it be that through working on the scene (adjusting daylight/material balancing etc) I then am running out of memory and have to relaunch cinema to get my 3.5gb back?
AW

This does not directly help your situation, but it sounds like you have what is called a memory leak, and understanding what that is might be useful. Essentially, as data gets moved around, written, read, deleted, sometimes the software loses track of little bits of free space, recording it as in use or otherwise unavailable, when it actually should be available. This often happens in very small pieces, but the longer you work, the more operations you do, and the more times memory locations get written and erased, thousands of chunks of data get shuffled around, those little small errors add up. Eventually, the amount of free memory drops below the threshold required for proper operation, and crashes, odd behaviors, etc. kick in.

There can be many layers of abstraction in the code managing memory, from firmware, to drivers, to OS, to application, and even multiple layers within those, so tracking these kinds of errors down can be especially tricky.

https://en.wikipedia.org/wiki/Memory_leak

Thanks for your reply A few days ago this is exactly what I though it could be, which is why I was questioning if it could be plugin side as it only exists inside C4D/Octane. Also I ruled hardware/windows out as I doubted that memory would be leaking simultaneously across 4 cards? But without knowing enough about it I wasn't sure if this was possible...

[glimpse]

we'll point this to devs, maybe there is a way to flush things out and keep the full amount under Your disposal.

in any case, thanks for reporting, I guess the hard part (for devs will be) will be to figure out what went wrong..

[frankmci]

Thanks for your reply A few days ago this is exactly what I though it could be, which is why I was questioning if it could be plugin side as it only exists inside C4D/Octane. Also I ruled hardware/windows out as I doubted that memory would be leaking simultaneously across 4 cards? But without knowing enough about it I wasn't sure if this was possible...

Yes, I didn't mean to imply that your problem is not Octane plugin or render engine related, just that these sorts of things can be tricky to track down.

That's why I mentioned the many layers of abstraction involved. It can be that it only happens with a particular combination of things, even if each individual element is performing as designed and as expected. Test any given element, and everything works great, test a bunch of permutations on a set of elements, and maybe they all work, or maybe, in certain circumstances, there are unforeseen consequences. So, for instance, it could be that the plugin is working as intended when working with a particular driver or with a particular firmware or with a particular hardware configuration, but the combination of all of those things causes problems, even though every part is working as expected. So where exactly is the bug? Where to point the finger? Whose design specs should change? Usually, whoever has the smallest customer base and the least money gets stuck with having to accommodate everyone else. In this case, it's probably Otoy that has to fix the problem.

Of course, it could just be a pain old bug, too.

[AaronWestwood]

Thanks for your reply A few days ago this is exactly what I though it could be, which is why I was questioning if it could be plugin side as it only exists inside C4D/Octane. Also I ruled hardware/windows out as I doubted that memory would be leaking simultaneously across 4 cards? But without knowing enough about it I wasn't sure if this was possible...

Yes, I didn't mean to imply that your problem is not Octane plugin or render engine related, just that these sorts of things can be tricky to track down.

That's why I mentioned the many layers of abstraction involved. It can be that it only happens with a particular combination of things, even if each individual element is performing as designed and as expected. Test any given element, and everything works great, test a bunch of permutations on a set of elements, and maybe they all work, or maybe, in certain circumstances, there are unforeseen consequences. So, for instance, it could be that the plugin is working as intended when working with a particular driver or with a particular firmware or with a particular hardware configuration, but the combination of all of those things causes problems, even though every part is working as expected. So where exactly is the bug? Where to point the finger? Whose design specs should change? Usually, whoever has the smallest customer base and the least money gets stuck with having to accommodate everyone else. In this case, it's probably Otoy that has to fix the problem.

Of course, it could just be a pain old bug, too.

100%. What you described in your initial reply describes exactly what im going through to a T. But also as you said, troubleshooting it is so so tricky!

Either way, my VRAM is going somewhere as I click around and tweak sliders. It IS a heavy scenes scattering close to 1 million clones. But its all very optimized and fits inside the VRAM upon each launch of cinema. Then it freaks out!