I think you can make a sphere surrounding the scene with the hdri projected and then make the object invisible for the camera+shadows!
Should work i think.
thanks for the idea
but how to hide that object for camera and shadows?
making it zero opacity hides it completely so it's not visible for rendering ...