Script a GUI item that doesn't appear in UI browers?

whitebalance · January 18, 2021, 11:33pm

Hello,

I’m trying to figure out how to find a GUI control point, assign a left click and hold command that can then be used by a Kensington trackball to move, and release after I press a button in the DaVinci Resolve application, Curves group.

I also may be approaching this wrong. Is there another way to do this? Python, JS Node, Command Line?

Is there a way to “browse” shell commands for an application the way UI browser’s identify scriptable UI elements?

The first part is relatively straightforward, as initiating the GUI control point is clearly visible in the four GUI inspectors I have access to, as you can see from screenshots below.

Initiate GUI control point:

https://drive.google.com/file/d/12PbbfbtLp3Tl-G3rl-EyU3yBDBwb0ZsO/view?usp=sharing

It looks like I could just use an applescript, or even display position and click, to engage the GUI control point by clicking the GUI control radio button within red box number 1 at the bottom. Once that is done, the application creates a second GUI control point inside the red box number 2. If I use a mouse, I can now click, hold, and drag that second GUI control point and move it around, affecting the Curves control in DaVinci Resolve.

How can instead create an applescript or command line shell to:

a. Select the first GUI control point (box 1)
b. Identify the second GUI control point (box 2) created after the selection of GUI control point 1.
c. Click and hold the second GUI control point (box 2) so that I can use a Kensington trackball to move it around.
d. Assign a button to release control of the second GUI control point.

You can see Xcode’s UI browser provides a performable action for GUI point 1 (box 1):

I can’t find an executable action listed in the four UI GUI inspectors for the second GUI control point (box 2), when I click on that second GUI control point:

Xcode UI browser:

https://drive.google.com/file/d/1QYA8ewqdkEWaejR67NnpIbdqiskXapAx/view?usp=sharing

The whole box is green, does that mean the whole box is treated as one selection, and nothing inside can be scripted?:

https://drive.google.com/file/d/16Cy_yFDy29QUpGnbsIFJzwcR-fmwVtf7/view?usp=sharing

Here are two macro UI identifiers generously provided on KM’s forum:

https://drive.google.com/file/d/1bC29qknCmHxTtklPJZ95gFM9P8jC-1UW/view?usp=sharing
https://drive.google.com/file/d/1CXE-05nU4QP0ci3jyrgBRtDncYsI_xpA/view?usp=sharing

Thanks!

KniazidisR · January 19, 2021, 4:57am

Open your interface
Open Automator, Choose Create Quick Action.
Click RED button (it is RECORD function of Automator, otherwise, its Follow Me function). It will record your clicks
perform in your inteface your actions a. b. c. d. manually
Stop Automator Recording

Now the Automator has recorded your manual actions. In Automator window see what script wrote Automator for you If UI element you need exist, it should be included in the recorded Automator’s script.

If it doesn’t exist, then you should use mouse tools, as suggested by Fredrik71

NOTE: to see Automator’s script, you should drag recorded actions to Automator’s code area, one by one

whitebalance · January 20, 2021, 1:07am

Hello, is mouseclick able to determine the x and y position of a GUI item on it’s own? The point is that the second GUI element is movable. Once it’s moved, how do I script a way of finding it again, selecting, so that I may move it again? It’s not a fixed GUI location. It can be anywhere.

Thanks!

whitebalance · January 21, 2021, 3:01am

Ooooh!!! I’m gonna give this a try.

I was trying with image recognition, and got close, but I could not move the control point once it was selected, even though I was using the “click and hold” action in Keyboard Maestro:

https://drive.google.com/file/d/1bp5dZbDc1E6-svI-SRbJcBnGwSfhQFCN/view?usp=sharing

https://drive.google.com/file/d/1Qp47pAPYgNbo8qvEg3tOuqXin_KBG9lS/view?usp=sharing

Thanks, can’t wait to try.

whitebalance · January 21, 2021, 3:05am

Wait, just to be clear, those terminal scripts will identify the GUI control point in the red box, numbered 2, not the entire window?:

https://drive.google.com/file/d/12PbbfbtLp3Tl-G3rl-EyU3yBDBwb0ZsO/view?usp=sharing

Thanks!

whitebalance · January 23, 2021, 5:43am

Thank you Fredrik 71…

Shell click makes it work! I did some testing, and just want to confirm that it’s the API that’s stopping me from using Applescript to click? I can get Applescript to click on other elements in the UI:

https://macscripter.net/viewtopic.php?pid=205332#p205332

Thanks again!!!