After several months with no releases I would like to give an update of the current development. Although it has been quiet, it doesn't mean there was no work going on. Quite the opposite: We are currently wrapping up the by far largest re-factoring in the history of Octane. It was quite painful task and I'm glad that most of it is over.
During the last months we experienced two setbacks that confirmed my opinion that we should not give any details about future plans and developments until they are almost done:
- The re-factoring of the node system took about twice as long as anticipated: I was originally hoping to have the next release out by the end of September, but looks like it will be this month (November). The main reasons are that we had to touch about 80-90% of the code base (i.e. everything with the exception of the core of the render engine) and that we found a lot more design issues and problems in the old system than we already knew.
- The new direct light sampler didn't work as well in real life scenes as in our test scenes: In our test scenes the image converged many times faster than with the old light sampler. We observed a reduction of the render time to get to the same noise level by about 50-80%, i.e. a speedup of 100-400%. But the new sampler introduced a lot of complexity, and GPUs don't like complexity. Not at all. As a result of the added complexity the overall sample rate slowed down a lot (about 10-40%, depending on the scene). But we had the faster convergence, right?
Well no... When we tested more real life architectural scenes, we experienced a bad surprise: Some areas in the image converged quite well, but some areas much worse than with the old sampler. We understand the problem, but we don't have a solution that doesn't introduce even more complexity. In the end we decided that this development was a dead-end and that we should cancel it. I.e. we will not release the new direct light sampler until we have found a better and faster solution.
We know it's disappointing, but be assured that we have tried really hard to get this working and it was quite frustrating to get stuck and not being able to make it work good enough in most cases. I think we will revisit this area not too far in the future.
Ok, the bad news out of the way, let's look at the positive things:
New Node System
Ok, some people might ask why go though the trouble and refactor the node system? The foundation of Octane is the node system. Pretty much every data is stored as nodes. Even the layout in the UI is stored as a node graph. The plugins build the scene by setting up a node graph. In other words: The node system is very important.
The basic design and idea of it is fine, but the actual implementation became more and more brittle. And every time we added some new feature we had to implement more and more hacks to get it somehow working. A few issues were:
- It was not possible to remove/replace pins. Ever noticed why the absorption medium node has strangely aligned pins? That's because there is a "hidden" pin, which is not used anymore, but could not be removed...
- It was not possible to re-order pins. Ever wondered why new pins always were added to the end of the list and why unimportant pins are at the top of the pin list?
- Pins had strange names and were sometimes camel case, had underscores or spaces.
- Slider ranges were derived from that last connected pin and stored in the node. Connection changes then didn't update the ranges properly.
- Many nodes had special functionality that was not exposed via the API. Which is why you haven't seen gradient support in the plugins yet.
- There was no animation capability.
- There was no system to convert from older versions to newer versions without hacking the OCS file reader.
- The event system was very complex and you easily forgot to send update events, which then caused lots of weird behaviour in Octane.
- ... and many many more ...
So the new node system is internally a full rewrite. On the outside it looks similar to the old one, but it's implementation is very different. Here a few visible improvements:
You can now create a node graph and hide some complex node set up. As in the old system, the interface to the "outside world" is done via input / output linker nodes, but now you can create multiple output linkers and input / output linkers can have now every pin type. Here for example, the node graph "Lights" contains three emitters and has three input linkers to control their power:
... all the rest is nicely tucked away inside:
In the node graph editor you can now select a group of nodes/node graphs and group them into a node graph with one click. You can then un-group that node graph with another click. Node graphs can be nested and copied. All macro nodes will be converted to generic node graphs. You may ask why all the effort? There were two main reasons:
- To allow plugins to hide the complex node setups they create in a node graph so that they will eventually be able to export a scene into an OCS file that can be opened and used in Octane standalone.
- Alembic files contain complex setups, that need to be loaded as complex node graphs. This will happen in a special node graph, which will have a geometry output linker and a camera output linker and input linkers for the various material groups.
The node system has now also a versioning system, that allows us to easier write converter code for older files, which already became important for the changes in the image texture nodes and the transformations. We have a completely new file format, which is a lot more compact, more readable (Still XML) and more flexible. Both changes also allow some forward compatibility: Octane now tries to load as much as possible from .ocs files, even if parts of it are unknown.
In the following screenshot you can see several more detail improvements:
- Cleaner UI.
- Up/down arrows at numeric edit fields (still require some tweaking, but we will get there...).
- Proper English naming of pins and nodes.
- Generic transform node, which adapts to the required dimension of the pin "through" which we are looking at it. It doesn't have input pins and thus allows a much more compact display.
- Replaced ListInt nodes with enum value nodes, which store a number and get their list from the pin "through" which we are looking at it.
- And of course, the texture transforms for image texture nodes itself, which is not a node system feature, but nice nonetheless (see below).
Image Texture Transforms
The scale float pin was replaced by a texture transform pin and a border mode was added:
I guess the options are self-explaining. Btw, the pin replacement and the required conversion of old scene files was what originally triggered the node system re-factoring.
You can now set some visibility properties for objects:
Here I have made the "spiky" washers to the left invisible to the camera. You see only its indirect effects, like shadows and caustics. You can also make it cast no direct shadows and you can set it's overall opacity, which can be useful for fade-in and fade-out effects.
What is an object, since mesh nodes store only a triangle soup? Each triangle can have an ID (number) assigned to, which can come from a plugin or from an OBJ file, which can objects/groups, too. You don't see the object IDs of a mesh node directly, but by connecting it with an object layer map node you can then assign an object layer node to the objects which you want to control.
In the image above you can also see the time slider. It's all still prototype-y, since we just started working on animation stuff a few days ago. We already had some stubs in the new node system, but now the actual work started and I expect it to progress fairly quickly. You can already scrub through the camera animation of an Alembic file. For 1.5 we aim to support loading animated geometry and camera data from Alembic files (with camera motion blur, without object/geometry motion blur). We don't plan to add animation editing capabilities to Octane in the near future, since this is a big task and we think we need to focus on the rendering features.
We support now the import of Alembic files, which are loaded as a node graph. It was quite a bit of work, but there is not much else to say except that we will provide an interface in the API, to store a geometry archive as Alembic file, which will eventually allow the plugins to export an animated scene as Alembic file, which is our final goal.
The next release will have scripting support with Lua:
It's based on the API and thus allows us to write test scripts and test the API. It also will allow us and users to add some functionality to Octane Standalone without touching the actual code base. It's pretty much work in progress (and will be for a while) and we plan to extend the scripting interface as we go along.
We added support for storing a scene as package, i.e. saving an .ocs file plus all used external files into one archive. It replaces embedding images in .ocs files, but obviously the much more important application of it is to pack a whole scene and to move it around, for example to the cloud.
It will not be visible to the normal user, but eventually he/she will enjoy the new abilities provided by the plugin devs, made possible with the changes in the API. One goal of the node system re-factoring was to increase the commonality between the API and the internals of Octane. We have done that. As a result, we plan to provide pretty much all functionality that is available in Octane via the API to the plugins. That is already partially done, but we will work on that in the coming weeks.
Next week we hope to get out a first release of the SDK and a test build of Octane SA to the plugin devs, so that they can get familiar with the changes in the API and give some feedback. The next public release (it will have version number 1.21) is planned for some time this month. After that I expect regular updates until we reach stable 1.5. I also expect quite a few issues and bugs in the initial releases, since this was a massive change.
There is also - since quite a while - work going on for features after 1.5, but I will keep my mouth shut and only say that I think you will like the new stuff. My current guess is that the next post-1.5 release will arrive some time in the first months of next year. But this is still quite blurry, because there is still a lot to do.
Ok, that wraps it up. Now that we started mostly adding new features again, it feels better again. I guess, the take away should be: If we are quiet on the forum, then that's not because we are not doing anything or ignoring people, but because we are working hard and just don't have time or have not progressed far enough with some big chunk of work to say stuff publicly.