Memory Problem

Forums: Memory Problem
Houdini Integrated Plugin

Moderator: juanjgon

Memory Problem

Postby ginodauri » Tue Feb 18, 2020 12:24 pm

ginodauri Tue Feb 18, 2020 12:24 pm
Hi
I'm rendering some mesh 10 mil polys , and system memory consumption is HUGE , 24 gb !
In comparing with mantra 5.5 gb.
What is happening here?

Houdini 17
2019.1.1
ginodauri
Licensed Customer
Licensed Customer
 
Posts: 14
Joined: Thu Oct 17, 2019 6:58 pm

Re: Memory Problem

Postby juanjgon » Tue Feb 18, 2020 12:58 pm

juanjgon Tue Feb 18, 2020 12:58 pm
It is hard to say what could be the problem without taking a look at the scene. I'm not sure if 24 GB of RAM for a 10M polygon is too much if it has UV maps and so on.

A quick test here shows that Octane needs around 15GB of RAM to render a 10M quads object (that are 20M triangles) without any attribute. Be aware that the scene must be extracted to a new temp database (something that Mantra doesn't need at all) and later Octane could need a lot of RAM to build the acceleration structures, but finally the important number is the GPU memory used to render the scene.

In any case, if you what to share the scene, I could take a look at it here.

Thanks,
-Juanjo
Computer Graphics Software Developer
Octane plugin development and support: LightWave and Houdini
User avatar
juanjgon
Octane Plugin Developer
Octane Plugin Developer
 
Posts: 7662
Joined: Tue Jan 19, 2010 12:01 pm
Location: Spain

Re: Memory Problem

Postby ginodauri » Tue Feb 18, 2020 1:26 pm

ginodauri Tue Feb 18, 2020 1:26 pm
Ok , i understand that octane needs extra memory because of conversion process etc , but this is huge difference , it is like 5x!

Is it possible somehow to make this a little closer to mantra performance , exporting orbx , using standalone or something?

And what i should expect at the end , consumption of mantra system memory should equal gpu consumption memory?

This is mesh file:
https://we.tl/t-Vp8xEhZJxO

Octane render setting was on default
ginodauri
Licensed Customer
Licensed Customer
 
Posts: 14
Joined: Thu Oct 17, 2019 6:58 pm

Re: Memory Problem

Postby juanjgon » Tue Feb 18, 2020 1:44 pm

juanjgon Tue Feb 18, 2020 1:44 pm
Here I can see that Octane is using around 15GB of RAM itself for rendering, near all of it used by the Octane core to build the acceleration structures, AI scene cache, etc. Probably one of the reasons why Octane needs so much RAM is due to the small size of the final triangles. The object is made using quads, so finally, you have around 22.9M triangles and the final GPU memory used for rendering is 3.7GB, as you can see in the attached image.

I suppose that this is the expected Octane behavior, but in any case, even with 32GB of CPU RAM you should have no problem to render objects like that. Mantra probably doesn't build so complex acceleration and cache structures, but well, in the end, thanks to all this additional data Octane can be a lot faster ;)

Thanks,
-Juanjo
Attachments
2020_02_18-14_38_33.jpg
Computer Graphics Software Developer
Octane plugin development and support: LightWave and Houdini
User avatar
juanjgon
Octane Plugin Developer
Octane Plugin Developer
 
Posts: 7662
Joined: Tue Jan 19, 2010 12:01 pm
Location: Spain

Re: Memory Problem

Postby ginodauri » Tue Feb 18, 2020 1:55 pm

ginodauri Tue Feb 18, 2020 1:55 pm
Thanks Juanjo ,

But when i submit render on deadline it reports , 24 GB peak.
Is this because houdini also needs to keep copy of geometry in memory.
Is it possible to make this less memory intensive , with orbx , just to handle converted data to octane or something .

The thing is that i have beside that mesh additional elements , so in my full render it was 54 gb system memory , and octane crashed on 85 % gpu memory.
ginodauri
Licensed Customer
Licensed Customer
 
Posts: 14
Joined: Thu Oct 17, 2019 6:58 pm

Re: Memory Problem

Postby juanjgon » Tue Feb 18, 2020 2:08 pm

juanjgon Tue Feb 18, 2020 2:08 pm
Yes, Deadline launches an hython process that must load the Houdini scene to later render it, so probably the final memory needed is the same that you use while working with the scene in the main Houdini application. You could export the scene to .orbx files, but this is not going to help if your problem is the GPU memory ... this is going to be always the same.

What could help you, in this case, is enabling the out of core feature, that makes possible the rendering of Octane scenes that don't fit physically in the available GPU memory. This feature can be enabled in the plugin options panel or using the "Octane_setOOC" hscript command. I'm not sure if Deadline has this command exposed, but I suppose that you could add it to the rendering script in any case.

Hope it helps,
-Juanjo
Attachments
2020_02_18-15_07_29.jpg
Computer Graphics Software Developer
Octane plugin development and support: LightWave and Houdini
User avatar
juanjgon
Octane Plugin Developer
Octane Plugin Developer
 
Posts: 7662
Joined: Tue Jan 19, 2010 12:01 pm
Location: Spain

Re: Memory Problem

Postby ginodauri » Tue Feb 18, 2020 2:37 pm

ginodauri Tue Feb 18, 2020 2:37 pm
Thanks
If i export to orbx files , can that handle problem of system memory?
ginodauri
Licensed Customer
Licensed Customer
 
Posts: 14
Joined: Thu Oct 17, 2019 6:58 pm

Re: Memory Problem

Postby juanjgon » Tue Feb 18, 2020 3:50 pm

juanjgon Tue Feb 18, 2020 3:50 pm
Yes, probably the rendering of a .orbx file using the Standalone needs less CPU memory because the Houdini scene doesn't need to be loaded, but the used GPU memory is going to be the same.

Thanks,
-Juanjo
Computer Graphics Software Developer
Octane plugin development and support: LightWave and Houdini
User avatar
juanjgon
Octane Plugin Developer
Octane Plugin Developer
 
Posts: 7662
Joined: Tue Jan 19, 2010 12:01 pm
Location: Spain

Re: Memory Problem

Postby ginodauri » Wed Feb 19, 2020 1:36 am

ginodauri Wed Feb 19, 2020 1:36 am
Hi

I tried to export orbx file from houdini and render that file in octane standalone.
Situation is little better it user 22 gb , compared to houdini 24.5 gb.So we can probably concluded that overhead is because houdini hython , etc.
And correct me if i'm wrong but orbx exports alembic files for meshes , and do not save some internal mesh structure , that can lower memory or speed up render ,
so basically this is same as if you build your octane standalone scene with alembic mesh importer.

But i'm also thinking is system memory actually linked to gpu memory.Because if i compare in my system used vs free memory , cpu is 23/64 and gpu is 4/11 , so it is same ratio , so i'm thinking octane uses equal available cpu memory to build those accelaration structures as you said.So i if i would have 32 gb memory , system memory utilization would be 11 gb.
SO my thinking here can system memory be problematic for octane rendering?
ginodauri
Licensed Customer
Licensed Customer
 
Posts: 14
Joined: Thu Oct 17, 2019 6:58 pm

Re: Memory Problem

Postby juanjgon » Wed Feb 19, 2020 12:51 pm

juanjgon Wed Feb 19, 2020 12:51 pm
I think that the problem is that the memory used by the Octane core is not always the same for the same number of polygons or instances. It can be related to the objects and instancing topology. Probably a lot of small triangles near the same place, or a lot of instances also near the same location can need deeper BVH trees and so on ... in the other hand, the GPU memory used by the textures has nothing to do with the CPU RAM, so a lot of textures can fill the GPU RAM without taking so much CPU RAM. The used CPU RAM can also be related to the number of AOVs and the render resolution or the number of GPUs in the system ... it is hard to say.

A 64GB system should be enough to render over 11GB GPUs for sure without access to the system paging file ... probably 32GB is also enough, but it is hard to say.

Thanks,
-Juanjo
Computer Graphics Software Developer
Octane plugin development and support: LightWave and Houdini
User avatar
juanjgon
Octane Plugin Developer
Octane Plugin Developer
 
Posts: 7662
Joined: Tue Jan 19, 2010 12:01 pm
Location: Spain
Next

Return to Houdini


Who is online

Users browsing this forum: No registered users and 1 guest

Thu Jul 02, 2020 9:18 am [ UTC ]