I've looked into this scene, and I can explain what's going on.
Even though the scene file is only 10 GB on disk, loading it and preparing it to be used for rendering requires a lot of system RAM (about 54 GB). There are various data structures that need to be built (by the CPU) that simply take up a lot of space. With 32 GB of RAM, I can't even get past this step. With 64 GB of RAM it looks like you can, but that will be dependent on the other applications running (and/or the size of your pagefile).
Because the scene consists primarily of one huge mesh (100 million tris) rather than bunch of smaller meshes, this 54 GB needs to be allocated as one big chunk. It obviously can't fit into GPU memory, so it has to go into out-of-core memory, and 54 GB, being almost all your system RAM, is more than what the OS is willing to let you pagelock for the GPU to use (and more than the different limits you set in Octane, anyway).
Some things that could theoretically improve this:
- Modify the scene to split the large mesh into smaller meshes (so that the GPU's 8 GB can be used, reducing the amount of out-of-core memory required). This is probably not very practical in this situation.
- Octane could automatically split the large mesh into smaller meshes (same thing but automatic, this would be a pretty complex change though).
- Octane could use less memory when preparing the scene for rendering (I haven't investigated deeply into why it uses so much).
- Get more RAM (unclear whether this would help because the limit might not be a proportion of RAM; it could be a fixed number or based on VRAM for example).
- Set an even higher OOC limit in Octane (say, 60 GB), just to make sure that you are hitting the OS limit rather than Octane's limit (in this case you should not see the "Out-of-core memory limit exceeded" error - it will be a different error).
- Increase the OS pagefile size (very unclear whether this would help but it's an idea; might reduce pressure on system RAM enough to squeeze the OOC data in there).
More VRAM will not help (unless you find a GPU with 64 GB) because of the mesh being one single chunk - each mesh needs to fit entirely in VRAM or OOC. Really the elephant in the room is that third point - I will investigate if there's anything we can save there (but I assume it's already been pretty well optimized).
You could simply say that the fact that the FBX files don't work shows the benefit of the VIM files!