Out of core... just a good marketing strategy Tweet

[paride4331]

Hi guys,
could you try to cancel all vertex displacement map node, replacing them with a displacement map node please?
Let me hear from you.
Ragards
Paride

it looks like you found the guilty one ... I thought it was Vertex Displacement, but I forgot to delete a node in my tests.

But I'll keep testing, I am not sure if that's it.

I already commented on that... Otoy has to stop moving forward and make sure everything works... because it's frustrating to try to work, when you already have thousands of things to do and waste your time trying to find problems that shouldn't be there.

Hi samsue,
unfortunately no, I did other tests tonight and it doesn't depend on vertex map node only.
Regards
Paride

[jimho]

in this case I suppose crash depends on the RAM limit available on board.

I am sure it is not my case. there is 128G Ram on board and there is plenty of them free when octane crash.
when standalone is running there is available and unusable vram shown, between them is the head of room, when more than one octane is running (plugin or the new beta version standalone etc)the unusable part will increase and eat up the headroom and then crash,
this is what I see, the Ram on board is still a lot free.

[paride4331]

in this case I suppose crash depends on the RAM limit available on board.

I am sure it is not my case. there is 128G Ram on board and there is plenty of them free when octane crash.
when standalone is running there is available and unusable vram shown, between them is the head of room, when more than one octane is running (plugin or the new beta version standalone etc)the unusable part will increase and eat up the headroom and then crash,
this is what I see, the Ram on board is still a lot free.

Hi jimho,
ok thanks for your info.
Regards
Paride

[samsue]

Hi guys,
could you try to cancel all vertex displacement map node, replacing them with a displacement map node please?
Let me hear from you.
Ragards
Paride

it looks like you found the guilty one ... I thought it was Vertex Displacement, but I forgot to delete a node in my tests.

But I'll keep testing, I am not sure if that's it.

I already commented on that... Otoy has to stop moving forward and make sure everything works... because it's frustrating to try to work, when you already have thousands of things to do and waste your time trying to find problems that shouldn't be there.

Hi samsue,
unfortunately no, I did other tests tonight and it doesn't depend on vertex map node only.
Regards
Paride

I appreciate you doing more testing.

- resaved all the textures to a maximum of 4k
- disable vertex displacement

so far it's working, I'll find out when to put more assets on the scene.

[paride4331]

Hi samsue,
At this moment Beppe and me think it could depend on textures over 4k moved out-of-core, but it is difficult to say because out-of-core moves part of the scene according to the geometry and textures. In tyour case we can say that vertex displacement generating geometry causes Octane to move some textures to out-of-core and this generates the cuda error 2; But it's rather empirical, because it depends on the amount of vram on board and how much is available at that moment, so each scene behaves differently.
Regards
Paride

[samsue]

Hi samsue,
At this moment Beppe and me think it could depend on textures over 4k moved out-of-core, but it is difficult to say because out-of-core moves part of the scene according to the geometry and textures. In tyour case we can say that vertex displacement generating geometry causes Octane to move some textures to out-of-core and this generates the cuda error 2; But it's rather empirical, because it depends on the amount of vram on board and how much is available at that moment, so each scene behaves differently.
Regards
Paride

That makes total sense. and why would the GPU Head room be connected to that problem?

I copied the same tree several times, when it reached 40 million polys it also started to give error.

From what I follow the forum, a lot of people come with this problem, for me it's all related. If you get all the posts, that people send gpu error etc ... to me everyone has a link to this problem. Octane has a serious problem that is generating this instability.

When will the Nvlink be available for Gtx? (so you can use the vram of both video cards together)

[Lewis]

I think this out of core crashes started after v4.x octane and that's just about time when geometry/polys were getting into out of core option.
In Octane v4.x if i remember good it was only textures that could be used with out of core and i've not had crashes with OOC while in 2019.x and 2020.x they are often and unpredictable.

Maybe it would be good to split those two options as separate settings ?

1. Out of core for textures
2. Out of core for geometry

That way user could transfer only textures into OOC if he wants and leave VRAM for geometry. Or both if needed.

I sent some of my testing (simple BOX cube subdivided with 8k textures applied) to Juanjo to send them to Octane DEV team but in egnerla OOC is very very slow when it reaches above 3 GB of OOC memory. 1-2 GB it's OKish, slower to render than with VRAM but once it goes over 3GB it's massive, massive slowdown like 10x slower than with 1 GB OOC memory.

Here is my results from recent test sin Octane 2020.XB3 (YES i deliberately increased GPU headroom so i can force it to use more OOC but it's for test purposes since scene is not very big in general)


Test 1 (4*2080 Ti) - Denoiser ON, Parallel samples=16, Max Tile samples=32
_____________________
GPU headroom = 300 MB
Out of Core = 0 MB
Rendertime = 1m 8sec (68sec)


Test 2 (4*2080 Ti) - Denoiser ON, Parallel samples=32, Max Tile samples=64
_____________________
GPU headroom = 300 MB
Out of Core = 768 MB
Rendertime = 1m 18sec (78sec) = 15% slower


Test 3 (4*2080 Ti) - Denoiser ON, Parallel samples=32, Max Tile samples=64
_____________________
GPU headroom = 512 MB
Out of Core = 1024 MB
Rendertime = 1m 32sec (92sec) = 35% slower


Test 4 (4*2080 Ti) - Denoiser ON, Parallel samples=32, Max Tile samples=64
_____________________
GPU headroom = 1024 MB
Out of Core = 1792 MB
Rendertime = 2m 04sec (124 sec) = 82% slower


Test 5 (4*2080 Ti) - Denoiser ON, Parallel samples=32, Max Tile samples=64
_____________________
GPU headroom = 2048 MB
Out of Core = 2816 MB
Rendertime = 2m 57sec (177 sec) = 160% slower


Test 6 (4*2080 Ti) - Denoiser ON, Parallel samples=32, Max Tile samples=64
_____________________
GPU headroom = 4096 MB
Out of Core = 4989 MB
Rendertime = 16m 03sec (963 sec) = 1315% slower

[jimho]

I did another try on the out of core issue.

When I am running one octane standalone (version 2020 xb3), Rendering, it shows : Vram used 3799MB, available: 9184MB, OOC:0

As mentioned if there is another program that use Vram, conflict will happen and octane crash,

I opened another octane standalone (version 2019.1.5) loading a scene named occ.ocs which contains 14 models of people (bought from AXYZ or Render people etc studios),
each model contains 2 or 3 maps (4K or 8K texture each map)

for the second session polys are not very high (around 500K polys totally, 40k each) yet the textures are huge, when loading the occ.ocs octane 2019 does take out of core memory,
Interestingly, when finished loading, it is the first octane(2020) session crashed,while it was rendering!

the second session reamains alive, which contains 4k,8k textures and out of core,

this little test is repeatable and will make the crash every time,

So, My guessing is that the crashing issue may not related to texture size but related to the memory management,
and timing, that is why GPU head room is important, perhaps when the headroom is not cleared by octane in time, other applications try to use VRAM, conflict happens.

[samsue]

Wow, make a total sense of what you guys said.

[jimho]

I supose when octane is moving data between Vram and system memory there is need of buffer space in Vram, that maybe the headroom for,
if your scene contains huge data block which have to move together, means model cannot be seperated or 4k 8k map, means they have to be move at the same time,
In this case larger buffer is needed, and when this moving happens there is another application try to use the VRAM the headroom is squeezed.
so seems how to plan the size for the headroom is the key at the moment.

but anyway we user do not have idea what kind of situation is really happened with the data and the programe, ram etc,
evanutally it still should be Otoy to work out a mechanism which automatically manange the VRAM related issue.

Out of core as the general idea is working with no doubt, this is great and key improvement for octane, just need to refine in the details.