Materials thoughts ! (Long Topic but delicious Cookies)

[Sam]

Good morning everyone and welcome
please take a seat and some cookies

*lights fades off*

Here's a little brainstorming of me about Materials and how external renderers manage them.

Radiance in another thread:
the most questions you ask, the longer it will take me to get the demo download ready

So Radiance read all this only once and get back to coding please
Also ive writen that by not knowing all the Octane features. Its only a concept.
And Octane is maybe already doing some of these things. Im waiting for the Beta like everyone else.

Materials library and Presets are very important
A big part of any renderer is the ability to share and distribute Materials / Collection of Materials / Basic Presets to other users.
For example Maxwell got an online library of almost 4000 Materials !! (3782 actually)
Fryrender got only 400 Materials but its also online like Maxwell.
Modo also got a large library not only including Materials but also Models and Lights and even Skies.
Some 3D Package dont have online library but ship with Presets or got Fan-made online library.


Proprietary file formats = BAD
Maxwell got almost 4000 Materials online that's freaking crazy and AMAZING
Maxwell materials are in .mxm file format wich is Proprietary (WHAT THE **** ?!)

So for example you are browsing Maxwell library and see THE PERFECT material you need.
its just between that awesome diamond material and that beautifull brushed tungsten.
The exact color, exact size, exact everything... You think "wow I could save hours of works"
But suddently all the room around you shatter and your head explode...
Because you forgot you are using Mental Ray

Yep 4000 materials including very good ones but its only for Maxwell... ONLY
That suck so much you can say it. So yes proprietary file format is BAD
Its like DRM but in for 3D file formats !


How Octane Materials could rule the world
By being open with the world and the users. That's as simple as that.
They could probably also rule the world by packing one Cookie per Material


Octane can already read OBJ
That's great ! But what if you make change inside Octane you like to save out ?
You get back to your 3D Package and write everything again ? NO
Octane gonna export materials that's sure.


Octane gonna export Materials
That's great ! But scroll up a little bit and read again the "Proprietary file formats = BAD" chapter.
Yes exporting Materials is not everything. You need to export it well. And not Max-well


Open structure rule the world
So how does Octane should export materials you ask me. Here's my little vision of things.
Its not ultra-detailed but its the main idea of keeping Octane "open" to other 3D Packages.


An example called "CarbonFiber"
"CarbonFiber is beautifull men. He enjoy life because he's pretty. That's totally related because he was born in Octane.
He's the first CarbonFiber ever born in Octane actually. CarbonFiber is now Adult and ready for the outside world.
He got two children called CarPaint0345 and BrushedMetalGlossy but forget them they are only childs for now"

Ok we got CarbonFiber Material done in Octane, he's ready, he got all his textures, all his parameters. Its just a perfect material.
(like every material you gonna make in Octane)

Now we hit export and choose a destination directory like the Desktop
Let's look what happened in our Desktop

We got a new folder ! AMAZING ! Its called carbonfiber_octane

Now inside carbonfiber_octane folder we got

carbonfiber_settings.xml (its a xml file)
carbonfiber_settings_clear.xml (its another xml file)
carbonfiber_settings_nodes.xml (its another xml file again)
carbonfiber_color.jpg (color map, can be another image file format)
carbonfiber_bump.jpg (bump map, can be another image file format)
carbonfiber_aniso.jpg (anisotropic map, can be another image file format)
carbonfiber_normal.jpg (normal map, can be another image file format)
carbonfiber_reflection.jpg (reflection map, can be another image file format)
carbonfiber_preview.jpg (preview, can be another image file format or a picture of a cookie)

That's it. CarbonFiber is now sitting in our Desktop looking at the wallpapers.
Let's see what are the benefits of this structure now.

All inside a single folder
All is inside a folder. Every Operating System can see that folder. Every 3D Package can browse that folder.
Its easy to view the images inside. Its easy to store to external backup. You can .zip it or .rar it and share materials.
The folder got a _octane suffix so Octane know its related and can browse directory only showing these folders.


The Octane specific settings are in XML
Carbonfiber_settings.xml is a file that Octane read and write. Its all the settings of CarbonFiber in one file.
Its small (like a cookie). You can open it and edit it. You make translators to other 3D Package more easily.
For example anyone could write a "Octane to Vray" software wich read the Octane XML and write a Vray file.
All textures paths are also included in this XML and updated by Octane when the material got saved.
Read about XML in wikipedia


There's also a Human Readable version of this XML
Carbonfiber_settings_clear.xml is the same XML than Carbonfiber_settings.xml but readable by humans.
That mean everything is stripped down to the minimum. And formated to a readable thing with full english words and not lines of CODE.
It can be naturally read by humans. Its not used by Octane to read parameters but Octane update it like the other XML.
Anyone can open the XML and read the settings. Can be used to manually translate materials. You can quickly check how a material is made.
It can be called by an online Material library .HTML page to display the materials settings without downloading the Material itself.


There's even support for Octane Nodes
Carbonfiber_settings_nodes.xml is a file that Octane read and write. Its all the settings of Nodes that modify the Material CarbonFiber.
Its small (like all the cookies). You can open it and text edit it. You make translators also but it will be harder since not all software are node based.
Its not always used, only for very complex materials. When you open your material in Octane it ask you if you want to also import the Nodes settings.


Textures maps are in native format
Maps can be browsed like normal images. They got a suffix for Octane to know wich one is the correct file.
Support for any number of the same type map (multiple layers) (carbonfiber_color0001.jpg / carbonfiber_color0002.jpg and so on)
You can edit the image with any image editor. Octane can maybe look for changes in the materials currently opened in the Scene or at least have a reload all materials button.
You can mix different images formats and even change the image type (from jpeg to targa) with your scene opened.
Octane look for NAME_SUFFIX(NUMBER) and doesn't care about the file type. (well only with compatible file type at least)


There's even a preview !
The preview can be anything. Not only a regular Material test Scene like a sphere. A teapot. A cookie with baked normal maps tasty
Octane can generate that preview. Octane can read that preview so you dont need to make a render.
You can even put a watermark in this file if you want. That's your choice.


Backward compatible for ever !
In the XML header there's informations about all variables used and the Octane version number
Also all variables got parameters AND active/inactive status (for example bump_ammount ; active ; -20.45 ; 3)
That means all materials gonna be backward compatible with all older versions of Octane !

In depth example:

You got old materials made with Octane v4.6 and you open them with Octane v4.7
Problem is that Octane v4.7 introduce a new parameter (let say "Specularity_Sharpening") Its only a dummy example

Octane v4.7 gonna read the XML header from Octane v.4.6 and gonna see that its an old header (4.7 is bigger than > 4.6)
Octane gonna edit the XML and add the new Specularity_Sharpening variable to the XML with a default value and make it inactive
Now you got you material updated for Octane v4.7

Now you close v4.7 and re-open v4.6
Octane gonna read the XML header line by line and gonna see that its a new header (4.6 is lower than > 4.7)
Octane gonna look wich variables are new (bigger than v4.6) and not compatible with v4.6
Octane gonna ignore all these variables including the XML Specularity_Sharpening one



That's it, the end

OBJ (or .OBJ) is a geometry definition file format first developed by Wavefront Technologies for its Advanced Visualizer animation package. The file format is open and has been adopted by other 3D graphics application vendors. For the most part it is a universally accepted format. en.wikipedia.org

In 4-5 years I want to go to Wikipedia and read

Octane Open XML (or .XML) is a material definition file format based on the Standard Generalized Markup Language (.XML) first used by Refractive Software for its Advanced Unbiased GPU renderer package Octane. The file format is open and has been adopted by other 3D graphics application vendors. For the most part it is a universally accepted format. en.wikipedia.org

Yep im done. Write a reply if you have something to say

Thanks for listening me, Sincerely Sam

*lights back on stage*

[pixelrush]

I think we are getting way ahead of the game Sam.
We want @radiance to sleep and eat sometime

Lets give him a chance to get the beta out the door before we overrun him with enthusiatic suggestions.
I am sure he will pick up and implement stuff as time goes by.

[N30N]

Hi Sam,

I don't think it's very practical but if you really want to see this become a reality you should write up a draft specification. Once it exists then the various application developers could consider supporting it.

There does seem to be some redundancy in the directory/file layout, I'd suggest…

• Use a single human readable XML file.
• Drop the material underscore file prefix.
• tar the directory for easy sharing/transfer.
• Have a .omf.tar (open material format) suffix (instead of underscore octane).
• Number images by default eg. bump_01.jpg

[Sam]

@N30N

I agree on all of this, but I have no technical background at all.
Im just speaking about ideas and not how we name files (it can be anything)
The basic idea is to not follow the other external renderer and their proprietary file format.

.tar is good but does every OS can open it ?
A single human readable XML would be perfect but that's more work to do for Octane to read it correctly.

Just another benefit of the XML is for Renderfarms
The textures are moved to each renderfarms (take time and bandwidth) and after that only the XML change.
The XML is very small so if you dont change textures but only parameters its very fast to update across all the workstations.

[radiance]

Hi,

first of all thanks for spending all the time to write up that large post,
i'm gonna keep it noted and will refer to it once i work in this area to see if i can get some good ideas out of it.

Currently i'm working on other areas so i'd like to focus my concentration there first.

Radiance

[N30N]

.tar is good but does every OS can open it ?

Yep, there are tools/libs for all OSes. It's a very common way of concatenating files and is faster than using zip and the like.

A single human readable XML would be perfect

I have no problem with XML but using JSON may help the adoption (XML has a bit of a bad reputation).

[TZORG]

mxm describes the material in the way it exists internally in the Maxwell render engine. If you want Maxwell to read/write something else then either
1) Maxwell needs to somehow translate a format that is unrelated to the Maxwell material system
or
2) Maxwell needs to abandon its proprietary internal material system

#2 won't happen (because you are only proposing a file format, not a material system)

I don't understand the point about openness and human-readability. Do you think that if the Octane material format is human-readable then other apps will want to rely on this format? I think that would only work if Octane's format supports the expression of material concepts that aren't even meaningful within Octane

[Sam]

Yeah true but dont forget that big companies and studio have shader writers and they love open things.
Take the example of Nuke the compositing program, its 200% script-able so its a joy to use and modify to suit every workflows.

Even if nobody will use that open format it will help big companies to take Octane and use it in their workflow !
That's also why no big animation studio use Maxwell and Fryrender, its not always because of the speed its because its not suited and not enough open.

And if Octane re-define a file format for Unbiased materials maybe futur Unbiased renderers gonna use it because it would be already rock-solid and industry proof.
Im not speaking about already existing renderers like Maxwell who already has proprietary formats.
Like you said Maxwell not gonna abandon their format.

But look in 1-2 years if blender start to implement Unbiased render internally (and not using plugins) they gonna need to think about an open solution.
And I think with the increase of GPU computing there's a need for a GPU-Unbiased file format that is also open.

Finnally Octane is node based, and every node based software I saw are more open to scripts or plugins because of the internal structure of the software.
If Octane ship API to developers with "how-to code Octane nodes", everyone gonna make nodes. Octane staff cant make every people wishes and request.

Same thing if Octane introduce a shader system, people gonna use it. And medium-large studio with R&D teams gonna write translators to include Octane in the pipeline.

The question to ask is: "Is there an industry standard today in the GPU Unbiased rendering market?" and the answer is no.
Even Fryrender with Arion gonna keep their proprietary materials.

I think there's an oportunity for Octane to take the lead in that way.

[STRESS]

Forget the Maxwell nonsense, I say you're better of by sticking with open standards like OpenSL:

http://code.google.com/p/openshadinglanguage/

[Sam]

Good link here STRESS
But I read

OSL has syntax similar to C, as well as other shading languages. However, it is specifically designed for advanced rendering algorithms and has features such as radiance closures, BSDFs, and deferred ray tracing as first-class concepts.

Its an open shading language for biased materials right?
What about unbiased specific settings like complex IOR, thin coat etc.. ?
Maybe its even harder to try to modify OSL for Octane render than writing a complete new shading language