The smoothing could work, but the problem can be that each big not smoothed polygon is made with a bunch of smaller coplanar polygons, so in this case the full smoothing is not possible because it only happens in the border of the big polygons. Perhaps Octane has this limitation while rendering VDBs ... It is hard to say without taking a look at the object.
Computer Graphics Software Developer
Octane plugin development and support: LightWave and Houdini