The problem is very big, since I use a lot of Render Layers in my workflow and where I work and having this issue is very sad.
When using Render Layers, edge sampling is completly messed up.
In beauty its all ok, but when using any pass (anything from beauty pass or rende layer pass) edges of objects are glitched.
Rendering on disk using exr or png doesnt solve problem (tried with different compressions and in different modes linear/tonemapped).
Here is examples images and video:
Video:
Main/Beauty:

Diffuse pass:

My info:
A. Windows 10
B. GTX 1080Ti or GTX 670 (happens on multiple machines)
C. RAM Capacity 32GB
D. Nvidia driver version = 398.11
E. OctaneRender Standalone version = 3.08.3 (3080300)
F. OctaneRender plugin version = 3.08.3
G. Host application version = Cinema4D R17.055