Page 2 of 2
Re: Finding correct settings for rendering layer masks
Posted: Mon Mar 18, 2019 11:49 pm
by SundancePioneer
niestudio wrote:Perhaps you can upload a cropped version of your photoshop file.
It looks to me like your mixup is there rather than your render.
Sure thing, here it is:

- alpha-fail_cropped-1.psd (33.83 MiB) Viewed 2114 times
I used the Beauty pass from Octane and the OCol infopass - i used the method explained here to make use of the Object Color Id pass as a layer mask in PS:
https://vimeo.com/119850717
If this is not the right method, i'd be thrilled to know what i need to do
Thanks so much for your help!
Greetings
Sundance
Re: Finding correct settings for rendering layer masks
Posted: Tue Mar 19, 2019 12:39 am
by niestudio
It looks like you're using the C4D save file rather than the render layers you have setup.
Have you tried those?
This is just two of the layers, but you should be able to add all you need.
They are grouped into one folder in PS, and then that folder is masked with the OCol Layers image data.
This is the basic setup I use, and I don't ever see that fringe like you are getting.
Perhaps there is a way to use the C4D Save file, but someone else would have to advise on that as I don't even keep that checked.
Re: Finding correct settings for rendering layer masks
Posted: Tue Mar 19, 2019 12:45 am
by SundancePioneer
niestudio wrote:It looks like you're using the C4D save file rather than the render layers you have setup.
Have you tried those?
This is just two of the layers, but you should be able to add all you need.
They are grouped into one folder in PS, and then that folder is masked with the OCol Layers image data.
That looks very good - what settings did you use to get the layers that way? Or did you set it up yourself in PS?
I actually used the files that were put out by Octane Render (render layer mask and Ocol, tried both) - i only had the C4D Save checked to get a flat beauty pass.
Did you render my C4D file without any changes to the settings?
Thanks for your kind help!
Greetings
Sundance
Re: Finding correct settings for rendering layer masks
Posted: Tue Mar 19, 2019 1:20 am
by niestudio
SundancePioneer wrote:
Did you render my C4D file without any changes to the settings?
Yes, Those were the layers rendered from the file as you uploaded it.
Re: Finding correct settings for rendering layer masks
Posted: Tue Mar 19, 2019 7:51 am
by aoktar
niestudio wrote:SundancePioneer wrote:
Did you render my C4D file without any changes to the settings?
Yes, Those were the layers rendered from the file as you uploaded it.
Thanks for pointing the correct way. Great help!
Re: Finding correct settings for rendering layer masks
Posted: Tue Mar 19, 2019 8:22 am
by SundancePioneer
niestudio wrote:
Yes, Those were the layers rendered from the file as you uploaded it.
Thanks indeed, i can see now where the error in my workflow was!
Did you use any bending method for the layers? Is there a guide to the correct layer stacking order and blending modes when working with multiple passes?
Thanks again for taking the time to look into it!

much appreciated!!
Regards
Sundance
Re: Finding correct settings for rendering layer masks
Posted: Tue Mar 19, 2019 1:23 pm
by niestudio
this is a great one (all of his are great)
https://www.youtube.com/watch?v=jBO7ywXpxCY
Re: Finding correct settings for rendering layer masks
Posted: Wed Mar 20, 2019 6:49 am
by SundancePioneer
Thank you for the link and for your great help on this matter
Kind regards
Sundance
Re: Finding correct settings for rendering layer masks
Posted: Thu Mar 21, 2019 1:17 pm
by SundancePioneer
niestudio wrote:
Yes, Those were the layers rendered from the file as you uploaded it.
Hi niestudio,
sorry to open this again and bother you, but i seem to have another/new problem in this matter now:
I rendered seperate passes (Refl. Direct, Refl., Diffuse, Beauty, Shadow) - when using the approach you suggested via your PSD file, the mask works absolutely fine for Refl. Direct and Reflection, but the Diffuse Channel within the layer group gives me a white fringe again.
Any idea what could cause this? Is the way i am merging the layers wrong? I just used "Passthrough" in PS like in your file, with the diffuse channel at the very bottom. The reflection passes mask fine, but the Diffuse pass gives me problems.
Any help would be much appreciated!
Thanks in advance
Sundance
Re: Finding correct settings for rendering layer masks
Posted: Thu Mar 21, 2019 3:35 pm
by niestudio
Hi Sundance,
Looking back at the file you uploaded, it looks like you have a couple other things working against you.
1. in your camera imager you have pre-mulitply alpha and Disable partial alpha ticked. This is giving you an aliased edge.
2. your white background is rendered
with the product in your current setup, and that is what you are seeing on the edge.
I'd suggest reading up on the
Render Layer system and using that method.
https://vimeo.com/119848103 - this is an old version of octane, so the settings look different, but the technique remains similar.
The photoshop end of it will be similar if not the same, but you'll get your object(s) rendered without the background.
If you don't want to use render layer, unticking those camera imager settings should get you a tighter (anti-aliased) edge, and from there you could use Select > Modify > Contract…(1px) in photoshop on your mask selection.