How to retrieve location of the user on visionOS

Our app needs the location of the current user. I was able to grant access and the authorization status is 4 (= when in use). Despite of that, retrieving the location fails at almost all times. It returns the error:

The operation couldn’t be completed. (kCLErrorDomain error 1.)

It happens in both the simulator and on the real device. On the simulator, I can sometimes trick the location to be detected by forcing a debug location in Xcode. But this does not work on the real device.

What might be the root cause of this behavior?