nvlink 3090's

[jwfvr]

hi - I have 2 3090's with an nvlink, seems to be working, but the render times become unacceptable when ooc kicks in - the render time goes from 10 min. to 10 hrs. I assumed there would be a slow down, but not like this - is there anything I should be doing to improve the speed when ooc is used? or do I need to disable ooc?
thanks.

5950x
128gigs ram
3090 (2) nvlink
modo 16

the 3090's are connected using riser cables to accomodate the nvlink spacing

[glimpse]

but the render times become unacceptable when ooc kicks in - the render time goes from 10 min. to 10 hrs

Good day,

there are two options, both are costly on their own, but what to choose depends on Your case:

* optimize the scene, spend time to get it into VRAM of single GPU and that's roughly 24 - 1.4(due to WDDM driver mode) and then some extra to have some headroom prevent from crashes that happen way more often if You are close to the edge.
or..
* spend money for A6000 or new ADA based GPU. At 5 or even 6k those ain't that cheap, but considering new ADA based GPU will be close to performance of 4090 (that's 2x 3090s) it not so bad of a deal even if it costs 7k

so the answer depends..if You have little to no time and do not want to optimize those scenes, then get Hardware. If You do have time and prefer not to spend money, then work on those scenes to get them smaller.

The main reason You see such a drop, I would have to guess, is because Out of Core will disable RTX boost. so if Your scene takes advantage of this technology, suddenly You start to render multiple times slower. That could be serious hit, especially as that boost could be anywhere from few % to 30x..

Another thing is interface between GPU and mobo - if You end up using OOC, the information is moved from VRAM to RAM, and that interface becomes a bottleneck. Not sure how much that influence Your issue, but If You want to avoid it, make sure Your GPU run at x8 or even x16 gen4 - some cables might knock Your connection speed to like few lanes of gen3 or older.. - double check that.

So that's nothing too surprising, but I bet it's annoying. If You need some help or tips, please let us know and we'll try to help.

[SSmolak]

but the render times become unacceptable when ooc kicks in - the render time goes from 10 min. to 10 hrs

The main reason You see such a drop, I would have to guess, is because Out of Core will disable RTX boost.

Out of core - yes, but NVlink not. NVlink memory pooling is slower by 1-5%max. If he get that huge slowdown this is because NVlink doesn't work as expected in Octane and work the same as Out of Core - this is bug. As I know NVlink memory pooling doesn't work for textures ( Octane limitation ) but only for geometry.

[SSmolak]

the 3090's are connected using riser cables to accomodate the nvlink spacing

These riser cables can limit NVlink bandwidth. Here is tool for testing : https://www.pugetsystems.com/support/gu ... s-10-1266/

nvlink bandwidth.jpg (6.63 KiB) Viewed 2283 times

[glimpse]

but the render times become unacceptable when ooc kicks in - the render time goes from 10 min. to 10 hrs

The main reason You see such a drop, I would have to guess, is because Out of Core will disable RTX boost.

Out of core - yes, but NVlink not. NVlink memory pooling is slower by 1-5%max. If he get that huge slowdown this is because NVlink doesn't work as expected in Octane and work the same as Out of Core - this is bug. As I know NVlink memory pooling doesn't work for textures ( Octane limitation ) but only for geometry.

Yes, If the scene is heavy in geo, NVLINK could be used, if it's heavy on textures, OOC kick in and RTX is disabled.

Memory pooling in OR works for what it was made. In other words just because You see original amount of VRAM being reported, does not mean that it does not work. In any case, GPU engine will try to keep information mirrored, stored in each GPU before it splits just because it's the fastest way to deal with it. So it's for Your own benefit. Once geometry heavy scene get bigger than amount of VRAM you have, You technically have pooling in action. Just because interface does not reflect it, it does not mean it's not working. If I write info in lower case or upercases the information in it will not chnage..

[glimpse]

the 3090's are connected using riser cables to accomodate the nvlink spacing

These riser cables can limit NVlink bandwidth. Here is tool for testing : https://www.pugetsystems.com/support/gu ... s-10-1266/

nvlink bandwidth.jpg

If cable is between GPU and mobo, it should bot limit anything as information is going through NVLINK bridge, but having lower speeds for GPU alone could have some influence..- I just haven't seen anything on even remotely simillar extent.

[jwfvr]

I have been doing a deep dive into this as I have a project that really needs it - I will be sharing more in the next few days but I would like to state - for future readers - neither OOC or Nvlink disables RTX acceleration. its a long held belief but it is not the case. RTX works with either scenario. - and - Nvlink will not work if OOC is enabled. I am currently doing testing on speed differences between these options, as you can use more than 2 cards if you are using OOC. I was fortunate enough to know someone with more knowledge than myself on this -

[glimpse]

Just enabling OOC does not disable RTX boost but if You step over of what the card has in terms of VRAM, then You RTX boost is lost, because in order to use it information should be on the card, not outside.

In any case, will throw a smaller card in upcoming days and test that again, but at least it was a case before, unless something has been changed in a meantime.

[glimpse]

Source: https://techgage.com/article/nvidia-gef ... rformance/

Sounds very much like Your situation, so I'm not sure where the claim comes from

its a long held belief but it is not the case

In any case, looking to Your findings.

[jwfvr]

yeah its interesting - I am not an expert by any means in this area, I am relatively new to Octane and all ears for new information. I don't want to speculate on the article but its basically the same as what I ran into. I am sure there are many variables, of which I am one, but the the massive drop off in speed prompted me to look into this - and I am really only sharing my experience to hopefully help someone else - instead of saying "long held belief" I should have said, "everyone is telling me RTX is disabled with OOC" but having done quite a bit of testing and going back and forth with some of the devs, as of right now, when correct settings are used, I no longer have the insane performance drop. its pretty easy to test actually, if you have a scene that requires OOC, you can enable and disable RTX in the device settings and see quite clearly there is a difference without it. Definitely there is a performance hit when using OOC, but from all that I am seeing that is not the same as the Non-RTX performance hit. what I can add to this is, and please note I am not saying this as objective fact - when I was doing these tests, I noticed that if some other event was triggered during the render, I would then get the massive performance drop - like 10 min to 10 hr kinda drop. one time this came from me opening task manager, another was a screen saver being activated. - I don't want to push any mis-information, I am sure I don't have the whole story, just relaying my experiences. in my case I am doing this testing with 2 3090 FE's and 1 4090 FE. - and just to note, 3 cards running with OOC is faster than 2 cards with NVlink. (but this of course depends on the cards being used) the experience that prompted my original post seems to have come from running out of system memory - when I had the scene fully loaded I was around 118GB of system ram being used. when I rendered it in standalone, far less system memory was needed and the render time was "normal". again, I am no expert, just sharing information given to me and my own experiences.

win 10
128 Ram
(2) 3090's
(1) 4090
5950x
latest Nvidia drivers
octane 2021.1