development update

Forums: development update
Generic forum to discuss Octane Render, post ideas and suggest improvements.
Forum rules
Please add your OS and Hardware Configuration in your signature, it makes it easier for us to help you analyze problems. Example: Win 7 64 | Geforce GTX680 | i7 3770 | 16GB

development update

Postby abstrax » Fri Nov 08, 2013 12:09 am

abstrax Fri Nov 08, 2013 12:09 am
Hi all,

After several months with no releases I would like to give an update of the current development. Although it has been quiet, it doesn't mean there was no work going on. Quite the opposite: We are currently wrapping up the by far largest re-factoring in the history of Octane. It was quite painful task and I'm glad that most of it is over.

During the last months we experienced two setbacks that confirmed my opinion that we should not give any details about future plans and developments until they are almost done:

  • The re-factoring of the node system took about twice as long as anticipated: I was originally hoping to have the next release out by the end of September, but looks like it will be this month (November). The main reasons are that we had to touch about 80-90% of the code base (i.e. everything with the exception of the core of the render engine) and that we found a lot more design issues and problems in the old system than we already knew.
  • The new direct light sampler didn't work as well in real life scenes as in our test scenes: In our test scenes the image converged many times faster than with the old light sampler. We observed a reduction of the render time to get to the same noise level by about 50-80%, i.e. a speedup of 100-400%. But the new sampler introduced a lot of complexity, and GPUs don't like complexity. Not at all. As a result of the added complexity the overall sample rate slowed down a lot (about 10-40%, depending on the scene). But we had the faster convergence, right?
    Well no... When we tested more real life architectural scenes, we experienced a bad surprise: Some areas in the image converged quite well, but some areas much worse than with the old sampler. We understand the problem, but we don't have a solution that doesn't introduce even more complexity. In the end we decided that this development was a dead-end and that we should cancel it. I.e. we will not release the new direct light sampler until we have found a better and faster solution.
    We know it's disappointing, but be assured that we have tried really hard to get this working and it was quite frustrating to get stuck and not being able to make it work good enough in most cases. I think we will revisit this area not too far in the future.

Ok, the bad news out of the way, let's look at the positive things:

New Node System

Ok, some people might ask why go though the trouble and refactor the node system? The foundation of Octane is the node system. Pretty much every data is stored as nodes. Even the layout in the UI is stored as a node graph. The plugins build the scene by setting up a node graph. In other words: The node system is very important.
The basic design and idea of it is fine, but the actual implementation became more and more brittle. And every time we added some new feature we had to implement more and more hacks to get it somehow working. A few issues were:

  • It was not possible to remove/replace pins. Ever noticed why the absorption medium node has strangely aligned pins? That's because there is a "hidden" pin, which is not used anymore, but could not be removed...
  • It was not possible to re-order pins. Ever wondered why new pins always were added to the end of the list and why unimportant pins are at the top of the pin list?
  • Pins had strange names and were sometimes camel case, had underscores or spaces.
  • Slider ranges were derived from that last connected pin and stored in the node. Connection changes then didn't update the ranges properly.
  • Many nodes had special functionality that was not exposed via the API. Which is why you haven't seen gradient support in the plugins yet.
  • There was no animation capability.
  • There was no system to convert from older versions to newer versions without hacking the OCS file reader.
  • The event system was very complex and you easily forgot to send update events, which then caused lots of weird behaviour in Octane.
  • ... and many many more ...

So the new node system is internally a full rewrite. On the outside it looks similar to the old one, but it's implementation is very different. Here a few visible improvements:

You can now create a node graph and hide some complex node set up. As in the old system, the interface to the "outside world" is done via input / output linker nodes, but now you can create multiple output linkers and input / output linkers can have now every pin type. Here for example, the node graph "Lights" contains three emitters and has three input linkers to control their power:
node_graph_1.png

... all the rest is nicely tucked away inside:
node_graph_2.png

In the node graph editor you can now select a group of nodes/node graphs and group them into a node graph with one click. You can then un-group that node graph with another click. Node graphs can be nested and copied. All macro nodes will be converted to generic node graphs. You may ask why all the effort? There were two main reasons:

  • To allow plugins to hide the complex node setups they create in a node graph so that they will eventually be able to export a scene into an OCS file that can be opened and used in Octane standalone.
  • Alembic files contain complex setups, that need to be loaded as complex node graphs. This will happen in a special node graph, which will have a geometry output linker and a camera output linker and input linkers for the various material groups.

The node system has now also a versioning system, that allows us to easier write converter code for older files, which already became important for the changes in the image texture nodes and the transformations. We have a completely new file format, which is a lot more compact, more readable (Still XML) and more flexible. Both changes also allow some forward compatibility: Octane now tries to load as much as possible from .ocs files, even if parts of it are unknown.

In the following screenshot you can see several more detail improvements:
node_inspector_1.png

  • Cleaner UI.
  • Up/down arrows at numeric edit fields (still require some tweaking, but we will get there...).
  • Proper English naming of pins and nodes.
  • Generic transform node, which adapts to the required dimension of the pin "through" which we are looking at it. It doesn't have input pins and thus allows a much more compact display.
  • Replaced ListInt nodes with enum value nodes, which store a number and get their list from the pin "through" which we are looking at it.
  • And of course, the texture transforms for image texture nodes itself, which is not a node system feature, but nice nonetheless (see below).

Image Texture Transforms

The scale float pin was replaced by a texture transform pin and a border mode was added:
texture_transform.png

I guess the options are self-explaining. Btw, the pin replacement and the required conversion of old scene files was what originally triggered the node system re-factoring.

Object Properties

You can now set some visibility properties for objects:
object_properties.png

Here I have made the "spiky" washers to the left invisible to the camera. You see only its indirect effects, like shadows and caustics. You can also make it cast no direct shadows and you can set it's overall opacity, which can be useful for fade-in and fade-out effects.
What is an object, since mesh nodes store only a triangle soup? Each triangle can have an ID (number) assigned to, which can come from a plugin or from an OBJ file, which can objects/groups, too. You don't see the object IDs of a mesh node directly, but by connecting it with an object layer map node you can then assign an object layer node to the objects which you want to control.

Time Slider

In the image above you can also see the time slider. It's all still prototype-y, since we just started working on animation stuff a few days ago. We already had some stubs in the new node system, but now the actual work started and I expect it to progress fairly quickly. You can already scrub through the camera animation of an Alembic file. For 1.5 we aim to support loading animated geometry and camera data from Alembic files (with camera motion blur, without object/geometry motion blur). We don't plan to add animation editing capabilities to Octane in the near future, since this is a big task and we think we need to focus on the rendering features.

Alembic Support

We support now the import of Alembic files, which are loaded as a node graph. It was quite a bit of work, but there is not much else to say except that we will provide an interface in the API, to store a geometry archive as Alembic file, which will eventually allow the plugins to export an animated scene as Alembic file, which is our final goal.

Lua Scripting

The next release will have scripting support with Lua:
lua.png

It's based on the API and thus allows us to write test scripts and test the API. It also will allow us and users to add some functionality to Octane Standalone without touching the actual code base. It's pretty much work in progress (and will be for a while) and we plan to extend the scripting interface as we go along.

Packaging

We added support for storing a scene as package, i.e. saving an .ocs file plus all used external files into one archive. It replaces embedding images in .ocs files, but obviously the much more important application of it is to pack a whole scene and to move it around, for example to the cloud.

Improved API

It will not be visible to the normal user, but eventually he/she will enjoy the new abilities provided by the plugin devs, made possible with the changes in the API. One goal of the node system re-factoring was to increase the commonality between the API and the internals of Octane. We have done that. As a result, we plan to provide pretty much all functionality that is available in Octane via the API to the plugins. That is already partially done, but we will work on that in the coming weeks.

Future Outlook

Next week we hope to get out a first release of the SDK and a test build of Octane SA to the plugin devs, so that they can get familiar with the changes in the API and give some feedback. The next public release (it will have version number 1.21) is planned for some time this month. After that I expect regular updates until we reach stable 1.5. I also expect quite a few issues and bugs in the initial releases, since this was a massive change.

There is also - since quite a while - work going on for features after 1.5, but I will keep my mouth shut and only say that I think you will like the new stuff. My current guess is that the next post-1.5 release will arrive some time in the first months of next year. But this is still quite blurry, because there is still a lot to do.

Ok, that wraps it up. Now that we started mostly adding new features again, it feels better again. I guess, the take away should be: If we are quiet on the forum, then that's not because we are not doing anything or ignoring people, but because we are working hard and just don't have time or have not progressed far enough with some big chunk of work to say stuff publicly.

Happy rendering,
Marcus
In theory there is no difference between theory and practice. In practice there is. - Yogi Berra
User avatar
abstrax
OctaneRender Team
OctaneRender Team
 
Posts: 5483
Joined: Tue May 18, 2010 11:01 am
Location: Auckland, New Zealand

Re: development update

Postby grimm » Fri Nov 08, 2013 12:37 am

grimm Fri Nov 08, 2013 12:37 am
Thanks Marcus, all of this sounds wonderful. :) Will you be able to use Lua for procedural textures?
Linux Mint 20 x64 | Nvidia GTX 980 4GB (displays) RTX 2070 8GB| Intel I7 5820K 3.8 Ghz | 32Gb Memory | Nvidia Driver 460.56
User avatar
grimm
Licensed Customer
Licensed Customer
 
Posts: 1321
Joined: Wed Jan 27, 2010 8:11 pm
Location: Spokane, Washington, USA

Re: development update

Postby stratified » Fri Nov 08, 2013 12:44 am

stratified Fri Nov 08, 2013 12:44 am
Hi,

You won't be able to create a procedural texture directly from Lua (at least not initially). You can generate procedural images from Lua and attach these to your materials as image textures.

cheers,
Thomas
User avatar
stratified
OctaneRender Team
OctaneRender Team
 
Posts: 945
Joined: Wed Aug 15, 2012 6:32 am
Location: Auckland, New Zealand

Re: development update

Postby grimm » Fri Nov 08, 2013 12:53 am

grimm Fri Nov 08, 2013 12:53 am
Thanks Thomas,

So initially Lua will be used as a pre/post processor before/after the render? Does Lua have access to the object data? It would be fun to see if you could do simple displacement with it.
Linux Mint 20 x64 | Nvidia GTX 980 4GB (displays) RTX 2070 8GB| Intel I7 5820K 3.8 Ghz | 32Gb Memory | Nvidia Driver 460.56
User avatar
grimm
Licensed Customer
Licensed Customer
 
Posts: 1321
Joined: Wed Jan 27, 2010 8:11 pm
Location: Spokane, Washington, USA

Re: development update

Postby stratified » Fri Nov 08, 2013 1:34 am

stratified Fri Nov 08, 2013 1:34 am
grimm wrote:Thanks Thomas,

So initially Lua will be used as a pre/post processor before/after the render? Does Lua have access to the object data? It would be fun to see if you could do simple displacement with it.


You can do a bit more than just use it as a pre/post processor. You can control rendering from Lua however it's not possible to for example write your own shaders that are used in the render kernels. You can access all the data available in the nodes (if that's what you mean by object data). Basically, you get the same capabilities from Octane as the 3d party plugins.

Manipulation of geometry (and creating your own geometry) is possible from Lua. So you could apply a displacement map to that geometry (before it's rendered, not while rendering) but rolling your own displacement mapping from Lua sounds like a daunting task :)

The idea is that users can use it to create their own tools on top of Octane that are not generic enough to warrant an implementation in Octane itself.

cheers,
Thomas
User avatar
stratified
OctaneRender Team
OctaneRender Team
 
Posts: 945
Joined: Wed Aug 15, 2012 6:32 am
Location: Auckland, New Zealand

Re: development update

Postby grimm » Fri Nov 08, 2013 2:04 am

grimm Fri Nov 08, 2013 2:04 am
Ok, sounds like a lot of fun. Can't wait to try it out. Thanks again Thomas.
Linux Mint 20 x64 | Nvidia GTX 980 4GB (displays) RTX 2070 8GB| Intel I7 5820K 3.8 Ghz | 32Gb Memory | Nvidia Driver 460.56
User avatar
grimm
Licensed Customer
Licensed Customer
 
Posts: 1321
Joined: Wed Jan 27, 2010 8:11 pm
Location: Spokane, Washington, USA

Re: development update

Postby p3taoctane » Fri Nov 08, 2013 4:51 am

p3taoctane Fri Nov 08, 2013 4:51 am
Great update... Knew you were eating crutches and jaffas and head down tail up...
Thanks ... Looking forward to node packing.... My window looks like a world wide network

Thanks

Peter
Windows 7 Pro_SP 1_64 bit_48 GB Ram_Intel Xeon X5660 2.80 GHZ x2_6 580GTX_1 Quadra 4800
p3taoctane
Licensed Customer
Licensed Customer
 
Posts: 1418
Joined: Mon Jan 25, 2010 12:53 am

Re: development update

Postby xxdanbrowne » Fri Nov 08, 2013 6:22 am

xxdanbrowne Fri Nov 08, 2013 6:22 am
Awesome work guys! Can't wait.

Now I just need to figure out how to come up with a few grand for a new computer. :D
xxdanbrowne
Licensed Customer
Licensed Customer
 
Posts: 503
Joined: Wed Feb 06, 2013 2:38 am

Re: development update

Postby suvakas » Fri Nov 08, 2013 8:03 am

suvakas Fri Nov 08, 2013 8:03 am
Just wondering..
Can't you add this new direct light sampler as an experimental kernel?
Could it still be useful for some of us who don't do much architectural work?

If it speeds up scenes like this car animation you've posted, then it can be an option for some cases...
http://www.youtube.com/watch?v=4k0sOCf- ... etAKlv1Z2A

Suv
User avatar
suvakas
Licensed Customer
Licensed Customer
 
Posts: 503
Joined: Sun Jun 12, 2011 10:26 am

Re: development update

Postby glimpse » Fri Nov 08, 2013 8:08 am

glimpse Fri Nov 08, 2013 8:08 am
WOW! Seems a Lot of Great Stuff is Coming! Thanks for Post and All the Work You're Doing!
User avatar
glimpse
Licensed Customer
Licensed Customer
 
Posts: 3715
Joined: Wed Jan 26, 2011 2:17 pm
Next

Return to General Discussion


Who is online

Users browsing this forum: Google [Bot] and 7 guests

Thu Mar 28, 2024 9:20 pm [ UTC ]