Memory Problem Tweet

[ginodauri]

Hi
I'm rendering some mesh 10 mil polys , and system memory consumption is HUGE , 24 gb !
In comparing with mantra 5.5 gb.
What is happening here?

Houdini 17
2019.1.1

[juanjgon]

It is hard to say what could be the problem without taking a look at the scene. I'm not sure if 24 GB of RAM for a 10M polygon is too much if it has UV maps and so on.

A quick test here shows that Octane needs around 15GB of RAM to render a 10M quads object (that are 20M triangles) without any attribute. Be aware that the scene must be extracted to a new temp database (something that Mantra doesn't need at all) and later Octane could need a lot of RAM to build the acceleration structures, but finally the important number is the GPU memory used to render the scene.

In any case, if you what to share the scene, I could take a look at it here.

Thanks,
-Juanjo

[ginodauri]

Ok , i understand that octane needs extra memory because of conversion process etc , but this is huge difference , it is like 5x!

Is it possible somehow to make this a little closer to mantra performance , exporting orbx , using standalone or something?

And what i should expect at the end , consumption of mantra system memory should equal gpu consumption memory?

This is mesh file:
https://we.tl/t-Vp8xEhZJxO

Octane render setting was on default

[juanjgon]

Here I can see that Octane is using around 15GB of RAM itself for rendering, near all of it used by the Octane core to build the acceleration structures, AI scene cache, etc. Probably one of the reasons why Octane needs so much RAM is due to the small size of the final triangles. The object is made using quads, so finally, you have around 22.9M triangles and the final GPU memory used for rendering is 3.7GB, as you can see in the attached image.

I suppose that this is the expected Octane behavior, but in any case, even with 32GB of CPU RAM you should have no problem to render objects like that. Mantra probably doesn't build so complex acceleration and cache structures, but well, in the end, thanks to all this additional data Octane can be a lot faster

Thanks,
-Juanjo

[ginodauri]

Thanks Juanjo ,

But when i submit render on deadline it reports , 24 GB peak.
Is this because houdini also needs to keep copy of geometry in memory.
Is it possible to make this less memory intensive , with orbx , just to handle converted data to octane or something .

The thing is that i have beside that mesh additional elements , so in my full render it was 54 gb system memory , and octane crashed on 85 % gpu memory.

[juanjgon]

Yes, Deadline launches an hython process that must load the Houdini scene to later render it, so probably the final memory needed is the same that you use while working with the scene in the main Houdini application. You could export the scene to .orbx files, but this is not going to help if your problem is the GPU memory ... this is going to be always the same.

What could help you, in this case, is enabling the out of core feature, that makes possible the rendering of Octane scenes that don't fit physically in the available GPU memory. This feature can be enabled in the plugin options panel or using the "Octane_setOOC" hscript command. I'm not sure if Deadline has this command exposed, but I suppose that you could add it to the rendering script in any case.

Hope it helps,
-Juanjo

[ginodauri]

Thanks
If i export to orbx files , can that handle problem of system memory?

[juanjgon]

Yes, probably the rendering of a .orbx file using the Standalone needs less CPU memory because the Houdini scene doesn't need to be loaded, but the used GPU memory is going to be the same.

Thanks,
-Juanjo

[ginodauri]

Hi

I tried to export orbx file from houdini and render that file in octane standalone.
Situation is little better it user 22 gb , compared to houdini 24.5 gb.So we can probably concluded that overhead is because houdini hython , etc.
And correct me if i'm wrong but orbx exports alembic files for meshes , and do not save some internal mesh structure , that can lower memory or speed up render ,
so basically this is same as if you build your octane standalone scene with alembic mesh importer.

But i'm also thinking is system memory actually linked to gpu memory.Because if i compare in my system used vs free memory , cpu is 23/64 and gpu is 4/11 , so it is same ratio , so i'm thinking octane uses equal available cpu memory to build those accelaration structures as you said.So i if i would have 32 gb memory , system memory utilization would be 11 gb.
SO my thinking here can system memory be problematic for octane rendering?

[juanjgon]

I think that the problem is that the memory used by the Octane core is not always the same for the same number of polygons or instances. It can be related to the objects and instancing topology. Probably a lot of small triangles near the same place, or a lot of instances also near the same location can need deeper BVH trees and so on ... in the other hand, the GPU memory used by the textures has nothing to do with the CPU RAM, so a lot of textures can fill the GPU RAM without taking so much CPU RAM. The used CPU RAM can also be related to the number of AOVs and the render resolution or the number of GPUs in the system ... it is hard to say.

A 64GB system should be enough to render over 11GB GPUs for sure without access to the system paging file ... probably 32GB is also enough, but it is hard to say.

Thanks,
-Juanjo