The Problem
When the device is in Portrait Up (default) orientation, the describe-(all|point) commands output the same coordinate space as the screen rotation (effectively no rotation or transposition). When the device is in any other orientation, such as landscape, the frame width/height and x/y coordinates describe the frame correctly, but if you want to tap or swipe, or even take a screenshot, you have to apply a rotation to the coordinates.
The Workaround
In one of my apps I added a sentinel to the accessibility layer that checks UIView.window?.windowScene?.interfaceOrientation and emits a stable orientation, and in my automations I extract the orientation from describe-all and use a rotation map for all my interactions. The reason I do this instead of describe-point hit testing is because when you take screenshots, hit testing something that crosses a centerline has a chance of detecting the wrong orientation, so rotating the screenshot occasionally does not work.
The Solution
In the IDB companion, exposing a JRPC command for acquiring the interfaceOrientation (macOS defaults to portraitUp, iOS uses true detection), idb describe-orientation, and perhaps a transpose/rotation map. This would make it so the public facing API does not need to change at all while offering a really convenient solution to the rotation problem.
I would be willing to draft a PR implementing this if anybody else is interested.
The Problem
When the device is in Portrait Up (default) orientation, the
describe-(all|point)commands output the same coordinate space as the screen rotation (effectively no rotation or transposition). When the device is in any other orientation, such as landscape, the frame width/height and x/y coordinates describe the frame correctly, but if you want to tap or swipe, or even take a screenshot, you have to apply a rotation to the coordinates.The Workaround
In one of my apps I added a sentinel to the accessibility layer that checks
UIView.window?.windowScene?.interfaceOrientationand emits a stable orientation, and in my automations I extract the orientation fromdescribe-alland use a rotation map for all my interactions. The reason I do this instead ofdescribe-pointhit testing is because when you take screenshots, hit testing something that crosses a centerline has a chance of detecting the wrong orientation, so rotating the screenshot occasionally does not work.The Solution
In the IDB companion, exposing a JRPC command for acquiring the interfaceOrientation (macOS defaults to portraitUp, iOS uses true detection),
idb describe-orientation, and perhaps a transpose/rotation map. This would make it so the public facing API does not need to change at all while offering a really convenient solution to the rotation problem.I would be willing to draft a PR implementing this if anybody else is interested.