OTOY Forums

I have a large scene, 10GB ORBX file.

I have 2 x 2080 with 8GB each. Since the file is so large, I added 55 GB of my 64 GB of RAM to be "OUT OF CORE" memory.

I am still getting a failure from both GPUS despite this. I am wondering if it is normal that 10GB file cannot fit into all this out of core memory and if there is anything else I can do to run this file with success?

Hi,
the orbx dimension does not give any info about VRAM consumption when the scene will be loaded in GPU, especially if the .orbx file is an animated file.
VRAM consumption depends mostly from Geometry, Displacement, Grayscale Textures, RGBA Textures, and their bit depth, loaded at that frame.
With RTX 2080 under Win 10 and monitor connected, you should have roughly 5/6GB of available VRAM.
Have you done everything possible to reduce the VRAM consumption, optimizing the scene as much as possible?
For example, you can resize/reduce the resolution of the textures that are seen from distance, and 32bit/16bit textures can be reduced to jpg/8bit, to save a lot of space.
Also setting the Imagetexture node to Float/Grayscale instead of Normal/RGBA, for all the Bump/Roughness/Metallic textures, and in general for all grayscale textures, can save up to 3x memory per texture.
You can also use texture compression, and reduce Parallel sampling, to reduce the VRAM consumption.
8/16GB of Out-of-core should be more than enough for huge scenes, and consider that you need RAM also for handling the scene and frame buffer, so you cannot think to use 55GB of Out-of-core, with 64GB of RAM.

ciao Beppe

Hi Bepe,

The scene contains no animation.

Have you done everything possible to reduce the VRAM consumption, optimizing the scene as much as possible?

This is not a typical scene, we actually want to highlight problematic geometry. The company I work for (VIMAEC.COM) specializes in super-optimizing the geometry of Revit files for rendering. I'm not sure if you know, but Revit (3d architecture software) exports terrible geometry to 3ds Max and it slows everything down. What we do is completely rebuild this geometry from scratch using 3D modelling best practice. So a FBX export may have 10 million triangles and be unworkable for renders, animation,s VR, AR, etc. Meanwhile that same model would come in around 2-4 million triangles if exported through VIM. We want to show that difference, which means we have to process the bad FBX model as is for comparison.

Here is an example:

VIM import into 3ds Max is 38,301,246 triangles and took 5 minutes to import - Results in a very responsive file



FBX import into 3ds Max is 100,049,099 triangles and took 15 hours to import - Results in unresponsive file



For the materials, I just used a single standard material to override everything as some random elements we transparent.

So far I've not had any issues rendering the VIM files, but the FBX files are such a pain and fail in most situations.

-

I will go through all your comments and try again. I appreciate the knowledge and wisdom!!

Ah ok, thanks for the clarification.
Definitely a different issue then.
Octane does not like degenerated polygons, so probably the issue is in the geometry node.
Please, before loading the FBX in Standalone, go to menu Windows/Generate Log window, and add it to your layout.
Then please, copy paste the error message, or share a screenshot, after trying to load the FBX version.

ciao Beppe

All it says is

Out-of-core memory limit exceeded
Out-of-core memory limit exceeded

I tries with 5G, 15GB, 30GB, 40GG and 50Gb of out of core memory.

I will PM you a link to the file if you want to look at it.

Here is a screenshot of the screen after failure:

SamuelAB wrote:All it says is

Out-of-core memory limit exceeded
Out-of-core memory limit exceeded

I tries with 5G, 15GB, 30GB, 40GG and 50Gb of out of core memory.

I will PM you a link to the file if you want to look at it.

Hi, I'm an Octane developer looking into improving the ability to use out-of-core memory in situations like this. Could you please PM me the scene? It would be great to get some more information about exactly why Octane is having a problem with this scene.

One thing to note is that the OOC limit you set in Octane preferences is just the maximum that Octane will try to allocate, but there is another limit imposed by the OS/GPU/drivers (it appears to be impossible to know what that limit is until you hit it). It's unlikely you'll be able to use 50 GB of out-of-core memory even if you set that as the limit in Octane. Improving the user messaging around this limit is one of the things I'm looking into.

Great news! Sending the file now in a PM.

I've looked into this scene, and I can explain what's going on.

Even though the scene file is only 10 GB on disk, loading it and preparing it to be used for rendering requires a lot of system RAM (about 54 GB). There are various data structures that need to be built (by the CPU) that simply take up a lot of space. With 32 GB of RAM, I can't even get past this step. With 64 GB of RAM it looks like you can, but that will be dependent on the other applications running (and/or the size of your pagefile).

Because the scene consists primarily of one huge mesh (100 million tris) rather than bunch of smaller meshes, this 54 GB needs to be allocated as one big chunk. It obviously can't fit into GPU memory, so it has to go into out-of-core memory, and 54 GB, being almost all your system RAM, is more than what the OS is willing to let you pagelock for the GPU to use (and more than the different limits you set in Octane, anyway).

Some things that could theoretically improve this:

• Modify the scene to split the large mesh into smaller meshes (so that the GPU's 8 GB can be used, reducing the amount of out-of-core memory required). This is probably not very practical in this situation.
• Octane could automatically split the large mesh into smaller meshes (same thing but automatic, this would be a pretty complex change though).
• Octane could use less memory when preparing the scene for rendering (I haven't investigated deeply into why it uses so much).
• Get more RAM (unclear whether this would help because the limit might not be a proportion of RAM; it could be a fixed number or based on VRAM for example).
• Set an even higher OOC limit in Octane (say, 60 GB), just to make sure that you are hitting the OS limit rather than Octane's limit (in this case you should not see the "Out-of-core memory limit exceeded" error - it will be a different error).
• Increase the OS pagefile size (very unclear whether this would help but it's an idea; might reduce pressure on system RAM enough to squeeze the OOC data in there).

More VRAM will not help (unless you find a GPU with 64 GB) because of the mesh being one single chunk - each mesh needs to fit entirely in VRAM or OOC. Really the elephant in the room is that third point - I will investigate if there's anything we can save there (but I assume it's already been pretty well optimized).

You could simply say that the fact that the FBX files don't work shows the benefit of the VIM files!

Hi Karu,

This is super helpful. I can simply split the mesh.

Usually for gaming and mobile VR, you want to have few pieces of many triangles to reduce draw-calls. It seems like it's almost the opposite here.

I will carefully re-read this, play with my page-file and modify the scene.

Thanks for the help!!

-

We know VIM is much faster and lighter than FBX, but we need to get that through to the customers in an instant graphic way

Very excited to try rendering this model locally again!