Random but regular crashes while exporting orbx files Tweet

[vime]

Description: While exporting from Houdini to ORBX a segfault happens.
Crashes happen occasionally but on a regular basis.
The error appears to be a NULL pointer possibly happening in an iterator,
modifying the underlying data structure while iterating.

Although the error happens in the linked liboctane.so to reproduce it probably the Houdini plugin is required, that's why it is posted here.

Stack trace is included, an additional gdb core dump (2.3GB) can be provided if requested,
then it would be possible to inspect further with the crashed state like this:

Code: Select all

gdb  /opt/hfs18.5.759/bin/hython-bin core.1752911

A liboctane.so with debug symbols could provide more details on what is happening.

----------------------------- ENVIRONMENT ---------------------------------------------
A. Operating System, including version (i.e. Win 7, OSX 10.11.2, Ubuntu 14.04, etc.)
Ubuntu 20.04.4 LTS

B. Graphics Card(s) model (i.e. GTX 580 - 3GB, TITAN, etc.)
GeForce RTX 2060 SUPER
C. RAM Capacity (i.e. 6 GB)
256GB
D. Nvidia driver version (i.e. 7.50, 7.5.22)
Driver Version: 470.129.06 CUDA Version: 11.4

E. OctaneRender Standalone version, if installed (i.e. 2.24.2, 2.23, etc.)
OctaneRender_Enterprise_2021_1_5_linux

F. OctaneRender plugin version (i.e. v2.25 - 2.21)
Octane_2021.1.4.3
G. Host application version, including build number if available (i.e. 3ds Max 2016 Build 18.0)
Houdini 18.5.759

---------------------------LOG OUTPUT-----------------------------------------------
...
17:51:31 INFOR: [orbx export] ------- Initializing scene exporter
[Octane] 17:51:31 INFOR: [orbx export] ------- Initializing scene exporter
17:51:31 INFOR: [orbx export] ------- ORBX filename: "/cache/temp/orbx/U37164_37_T30437_1_11097623.orbx"
[Octane] 17:51:31 INFOR: [orbx export] ------- ORBX filename: "/cache/temp/orbx/U37164_37_T30437_1_11097623.orbx"
17:51:31 INFOR: [orbx export] ------- Scene Exporter: Exporting frame 152
[Octane] 17:51:31 INFOR: [orbx export] ------- Scene Exporter: Exporting frame 152
17:51:31 INFOR: [orbx export] ------- Closing the scene exporter
[Octane] 17:51:31 INFOR: [orbx export] ------- Closing the scene exporter
....

Code: Select all

----------------------------- GDB STACKTRACE ----------------------------------------------------
Thread 1 "hython-bin" received signal SIGSEGV, Segmentation fault.
0x00007fffc7c3ad83 in std::_Rb_tree_increment(std::_Rb_tree_node_base const*) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
(gdb) bt
#0  0x00007fffc7c3ad83 in std::_Rb_tree_increment(std::_Rb_tree_node_base const*) ()
   from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#1  0x00007fff9e083a38 in ?? ()
   from /home/ubuntu/houdini18.5/Octane_2021.1.4.3_Houdini_18.5.759_Linux_gcc6.3.1/dso/../bin/liboctane.so
#2  0x00007fff9e0842cd in ?? ()
   from /home/ubuntu/houdini18.5/Octane_2021.1.4.3_Houdini_18.5.759_Linux_gcc6.3.1/dso/../bin/liboctane.so
#3  0x00007fff9e0855e1 in ?? ()
   from /home/ubuntu/houdini18.5/Octane_2021.1.4.3_Houdini_18.5.759_Linux_gcc6.3.1/dso/../bin/liboctane.so
#4  0x00007fff9e0942ba in ?? ()
   from /home/ubuntu/houdini18.5/Octane_2021.1.4.3_Houdini_18.5.759_Linux_gcc6.3.1/dso/../bin/liboctane.so
#5  0x00007fff9e289a0a in ?? ()
   from /home/ubuntu/houdini18.5/Octane_2021.1.4.3_Houdini_18.5.759_Linux_gcc6.3.1/dso/../bin/liboctane.so
#6  0x00007fffa918304e in HOctane_sceneExporterCloseAndExportOrbx(ROP_Octane*, int) ()
   from /home/ubuntu/houdini18.5/Octane_2021.1.4.3_Houdini_18.5.759_Linux_gcc6.3.1/dso/Houdini_Octane_18.5.759.so
#7  0x00007fffa91d9abb in ROP_Octane::renderFrame(double, UT_Interrupt*) ()
   from /home/ubuntu/houdini18.5/Octane_2021.1.4.3_Houdini_18.5.759_Linux_gcc6.3.1/dso/Houdini_Octane_18.5.759.so
#8  0x00007ffff07f7fd8 in ROP_Node::executeSingle(double, ROP_Verbose*, SOHO_IPRList*, ROP_SohoOutput*, bool) ()
   from /opt/hfs18.5.759/bin/../dsolib/libHoudiniOP4.so
#9  0x00007ffff07fd7f4 in ROP_RenderItem::render(ROP_Verbose*, int, int) const ()
   from /opt/hfs18.5.759/bin/../dsolib/libHoudiniOP4.so
#10 0x00007ffff08068d9 in ROP_RenderManager::renderOrder(ROP_RenderList const&, ROP_Verbose*) ()
   from /opt/hfs18.5.759/bin/../dsolib/libHoudiniOP4.so
#11 0x00007ffff08078e2 in ROP_RenderManager::render(ROP_Node*, double, double, double, bool, bool, bool, bool, bool, ROP_Verbose*) () from /opt/hfs18.5.759/bin/../dsolib/libHoudiniOP4.so
#12 0x00007ffff07eb779 in ROP_Node::renderCommand(ROP_RenderCmdParms&) () from /opt/hfs18.5.759/bin/../dsolib/libHoudiniOP4.so
#13 0x00007ffff152c762 in HOMF_RopNode::render(std::vector<double, std::allocator<double> > const&, std::vector<double, std::allocator<double> > const&, char const*, char const*, bool, int, bool, HOM_EnumValue&, bool, bool, bool, bool) ()
   from /opt/hfs18.5.759/bin/../dsolib/libHoudiniOPZ.so
...


[juanjgon]

Hi,

This problem is a crash on the exporter's core side, in the Octane core lib. Do you have this problem with any scenes? If not, it could be great to have the scene here to try to reproduce the problem.

Also, I've seen that you are working with the Octane_2021.1.4.3 plugin. It could be great to know if the same happens with Octane 2021.1.5 or even with the beta build of Octane 2022.1 to discard that the problem could be already fixed:
viewtopic.php?f=102&t=79815
viewtopic.php?f=102&t=79890

I'm not sure if the gdb core dump is going to be useful to the core devs to narrow down the source of this problem, it could help so feel free to send it to us, but not as much as having the scene. You can send all this information to me in personal message in the forum or by email to juanjo [at] genesisvisual.com

Thanks,
-Juanjo

[vime]

Hi,
thanks for your response.
We have recently updated to the most recent Octane and Octane Plugin version, at least that's what i thought. The version available on the download page is still Octane_2021.1.4.3, in the forum i now also found the version Octane_2021.1.5.0 which i will try to install soon, although we have to test this properly which will not happen until Friday now.

The scene file is a bit hard to replicate, it is a base scene that dynamically loads in a number of HDAs.
I think i am not free to share all the required hda files that would be necessary to render, however i think i can send you the base scene if that helps.

I think the best approach would be to use a liboctane.so with debug symbols to see what is happening prior to the last command in the stack trace, if you are free to provide me with a file with symbols i could do that, otherwise the best way would be the core dump i think. I am going to upload the core dump file (2,3GB compressed and 19GB uncompressed) so you can check it out.

As mentioned the error can not always be replicated but happens occasionally in the production environment.
I am not sure if it is always the same error, but we have a lot of segfaults when shutting down the process after completing the export job (possibly caused by double free of memory).

I am now going to upload the core dump and send you an email with the link.
I am happy to test this with other versions but as mentioned it is not always easy to reproduce, i was happy enough to catch it this time when running it with gdb

[juanjgon]

Did you have a chance to send us the core dump file? I'm not sure if we can provide a liboctane.so with debug symbols, so we should try to figure out what could be the problem from the core dump or from the scene itself.

Thanks,
-Juanjo

[vime]

Maybe the email did not work that you gave me, i sent you now also a PM with the link to the core dump.

Cheers,

Lutz

[juanjgon]

Nope, sorry, I had not received the email. I'll try to take a look at this problem with the information you sent me in the pm.

Thanks,
-Juanjo