OTOY Forums

Community Forums for OTOY Customers
https://render.otoy.com/forum/

CUDA errors all over the place :(

https://render.otoy.com/forum/viewtopic.php?t=79332

Page 1 of 1

CUDA errors all over the place :(

Posted: Fri Feb 18, 2022 1:27 pm
by lalalem
Hi all,

I'm completely lost. I keep getting these errors and I have no idea why. I desperately need some help!

System:

Windows 10
Octane Enterprise 2021.1.2
Cinema 4D R25.115
NVIDIA Studio Driver 511.65
4x RTX 3090 (not Nv link).
Started logging on 18.02.22 20:21:20

OctaneRender Enterprise 2021.1.2 (11000200)

OCIO config file:C:\Users\lalalem\Downloads\aces_1.2\config.ocio loaded.
VRAM used/free/max:0Kb/24Gb/24Gb Out-of-core used:0Kb RAM used:8.97Gb total:95.847Gb OpenGL free/total:0/0

FRAME:0 fps:30
FumeFx library initialisation error!
--------------------------- EXPORT LOG ---------------------------
Export materials time= 14812.937 ms
Collect objects time= 10.201 ms
Mesh creation time = 4678.346 ms.
CUDA error 2 on device 2: out of memory
-> failed to allocate device memory
device 2: disabling RTX acceleration as mesh data cannot be placed into VRAM
CUDA error 2 on device 1: out of memory
-> failed to allocate device memory
device 1: disabling RTX acceleration as mesh data cannot be placed into VRAM
CUDA error 2 on device 3: out of memory
-> failed to allocate device memory
CUDA error 2 on device -1: out of memory
-> failed to create temporary context
CUDA error 2 on device 0: out of memory
-> failed to allocate device array
CUDA error 2 on device 3: out of memory
-> failed to allocate device array
CUDA error 2 on device 1: out of memory
-> failed to allocate device memory
CUDA error 2 on device 3: out of memory
-> failed to allocate device array
CUDA error 2 on device 0: out of memory
-> failed to allocate device array
CUDA error 2 on device 2: out of memory
-> failed to allocate device memory
CUDA error 2 on device 3: out of memory
-> failed to allocate device array
CUDA error 2 on device 2: out of memory
-> failed to allocate device memory
CUDA error 2 on device 3: out of memory
-> failed to allocate device array
CUDA error 2 on device 1: out of memory
-> failed to allocate device memory
CUDA error 2 on device 0: out of memory
-> failed to allocate device array
CUDA error 2 on device 2: out of memory
-> failed to allocate device memory
CUDA error 2 on device 3: out of memory
-> failed to allocate device array
CUDA error 2 on device 0: out of memory
-> failed to allocate device array
CUDA error 2 on device 2: out of memory
-> failed to allocate device memory
CUDA error 2 on device 2: out of memory
-> failed to allocate device memory
CUDA error 2 on device 3: out of memory
-> failed to allocate device array
CUDA error 2 on device 0: out of memory
-> failed to allocate device array
CUDA error 2 on device 1: out of memory
-> failed to allocate device memory
CUDA error 2 on device 2: out of memory
-> failed to allocate device memory
CUDA error 2 on device 3: out of memory
-> failed to allocate device array
CUDA error 2 on device 0: out of memory
-> failed to allocate device array
CUDA error 2 on device 1: out of memory
-> failed to allocate device memory
denoiserThread0 : Initialization failed. Restart required
Check geometry subdivisons.
Picture viewer uses 'Render' subdivisions. But Live Viewer uses 'Editor' subdivision values.
Try to decrease 'render/editor' subdivison values.
CUDA error 2 on device 3: out of memory
-> failed to allocate device array
CUDA error 2 on device 2: out of memory
-> failed to allocate device memory
CUDA error 2 on device 3: out of memory
-> failed to allocate device array
CUDA error 2 on device 1: out of memory
-> failed to allocate device memory
CUDA error 2 on device 2: out of memory
-> failed to allocate device memory
CUDA error 2 on device 3: out of memory
-> failed to allocate device array
CUDA error 2 on device 2: out of memory
CUDA error 2 on device 3: out of memory
-> failed to allocate device memory
-> failed to allocate device array
CUDA error 2 on device 1: out of memory
-> failed to allocate device memory
CUDA error 2 on device 2: out of memory
-> failed to allocate device memory
denoiserThread2 : Initialization failed. Restart required
CUDA error 2 on device 1: out of memory
-> failed to allocate device memory
CUDA error 2 on device 1: out of memory
-> failed to allocate device array
CUDA error 2 on device 2: out of memory
-> failed to allocate device memory
CUDA error 2 on device 1: out of memory
-> failed to allocate device array
CUDA error 2 on device 2: out of memory
-> failed to allocate device array
CUDA error 2 on device 2: out of memory
-> failed to allocate device array
CUDA error 2 on device 2: out of memory
-> failed to allocate device array
CUDA error 2 on device 2: out of memory
-> failed to allocate device array
Unhandled exception in concurrent loop: bad allocation
device 0: invalid memory pointers passed
device 0: path tracing kernel failed
device 0: invalid memory pointers passed
device 0: path tracing kernel failed
Tonemapping the all passes tm:0.035
pass [Denoised beauty] ID:43 not found...
Displaying passes in tm=0.805
Passes saved in:0.006sec.
Break rendering...
could not request a tonemap because we don't have any renderthreads available.
Tonemapping the all passes tm:0.216
pass [Denoised beauty] ID:43 not found...
Displaying passes in tm=0.383
Passes saved in:0sec.
**** RENDER FINISHED **** stat=3

Re: CUDA errors all over the place :(

Posted: Fri Feb 18, 2022 2:39 pm
by aoktar
Help starts with you! Begin some investigation what causes this. Remove some parts until error stops, and decide it's about some triangle count or VRAM limitations or something specific on any materials.

Re: CUDA errors all over the place :(

Posted: Fri Feb 18, 2022 3:50 pm
by lalalem
Yes I know. But the scene works well in the live viewer. VRAM usage: 9Gb used. 10Gb Free. So it should work right ? But I can't render it out!

Sometimes I get this different error log :
Started logging on 18.02.22 22:46:44

OctaneRender Enterprise 2021.1.2 (11000200)

OCIO config file:C:\Users\lalalem\Downloads\aces_1.2\config.ocio loaded.
VRAM used/free/max:0Kb/24Gb/24Gb Out-of-core used:0Kb RAM used:7.606Gb total:95.847Gb OpenGL free/total:0/0
FumeFx library initialisation error!
--------------------------- EXPORT LOG ---------------------------
Tried to access pin via an invalid ID P_SUB_TYPE (703)
Tried to access null node pin
can't read int/enum from static pin 'unknown' (P_UNKNOWN, type PT_UNKNOWN) of node '' (NT_UNKNOWN, 00007FF8751D70D0)
Export materials time= 14848.043 ms
Collect objects time= 13.806 ms
Mesh creation time = 3129.879 ms.
Api update time: 7.386 ms.
VRAM used/free/max:9.785Gb/11.973Gb/24Gb Out-of-core used:0Kb RAM used:38.699Gb total:95.847Gb OpenGL free/total:0/0

FRAME:0 fps:30
--------------------------- EXPORT LOG ---------------------------
Export materials time= 46.564 ms
Collect objects time= 8.855 ms
Mesh creation time = 2852.656 ms.
CUDA error 2 on device 1: out of memory
-> failed to allocate device array
CUDA error 2 on device 1: out of memory
-> failed to allocate device array
denoiserThread2 : Initialization failed. Restart required
Check geometry subdivisons.
Picture viewer uses 'Render' subdivisions. But Live Viewer uses 'Editor' subdivision values.
Try to decrease 'render/editor' subdivison values.
Failed to create CUDA context
denoiserThread3 : Initialization failed. Restart required
CUDA error 2 on device -1: out of memory
-> failed to create temporary context
device 2: invalid memory pointers passed
device 2: path tracing kernel failed
device 3: invalid memory pointers passed
device 3: path tracing kernel failed
device 2: invalid memory pointers passed
device 2: path tracing kernel failed
device 3: invalid memory pointers passed
device 3: path tracing kernel failed
CUDA error 2 on device 1: out of memory
-> failed to allocate device array
CUDA error 2 on device 1: out of memory
-> failed to allocate device array
Failed to create CUDA context
Unhandled exception in concurrent loop: bad allocation
Tonemapping the all passes tm:51.532
Displaying passes in tm=7.723
Passes saved in:0.274sec.
Break rendering...
could not request a tonemap because we don't have any renderthreads available.
Tonemapping the all passes tm:0.192
pass [Denoised beauty] ID:43 not found...
Displaying passes in tm=0.407
Passes saved in:0sec.
**** RENDER FINISHED **** stat=3

Re: CUDA errors all over the place :(

Posted: Fri Feb 18, 2022 7:12 pm
by aoktar
Only differences between LV and PV that's triangle counts when it's sent to renderer. My advice is still valid.

Re: CUDA errors all over the place :(

Posted: Fri Feb 18, 2022 7:46 pm
by lalalem
Is there nothing else that could cause this, from looking at the error log ?

Re: CUDA errors all over the place :(

Posted: Fri Feb 18, 2022 9:26 pm
by aoktar
lalalem wrote:Is there nothing else that could cause this, from looking at the error log ?
Always there's other possibilities. Better to ask to support team, they are more used to solve these kind of issues.

Re: CUDA errors all over the place :(

Posted: Sat Feb 19, 2022 10:37 pm
by lalalem
Just in case someone comes here from Google in the future : I managed to get around this problem by using NVLink. This allowed me to have way more stuff in VRAM, and I was finally able to render! Also it seems the main problem was a tennis ball hair object that I had converted to splines. There were millions of tiny spline hairs with an Octane Tag "use as hair" on it. That was the biggest problem.
All times are UTC
Page 1 of 1

Powered by phpBB® Forum Software © phpBB Limited