Ok. I did more digging into this.
Take a look at the images below.
The rendered castle towers are symmetry applied object. The part in the middle where you see the gate is seperate object.
Main model is the left part.
Right part is the mirrored / symmetrical part of the object. The problem area is on the right mirrored (by symmetry modifier) part.
The bump displays just fine in the left part at all times.
If the light comes from left or right. The bump shadow and lights applies as if the bump was inverted.
(Bottom image) With the same setup, the difference is the main light is coming from top.
The bump seems like it displays as it should be in all places. But there is a minor difference same as the old lighting scenario, the bumps the left and right acts as flipped in the right hand side of the castle.
So I guess the problem lies in the bump map invert function is triggered (only on X axis) when using the symmetry modifier.
Please solve this.