Monday, March 1, 2021

#1 2021-01-18 05:33:08 pm

whitebalance
Member
Registered: 2021-01-18
Posts: 10

Script a GUI item that doesn't appear in UI browers?

Hello,

I'm trying to figure out how to find a GUI control point, assign a left click and hold command that can then be used by a Kensington trackball to move, and release after I press a button in the DaVinci Resolve application, Curves group.

I also may be approaching this wrong.  Is there another way to do this?  Python, JS Node, Command Line?

Is there a way to "browse" shell commands for an application the way UI browser's identify scriptable UI elements?

The first part is relatively straightforward, as initiating the GUI control point is clearly visible in the four GUI inspectors I have access to, as you can see from screenshots below.

1.  Initiate GUI control point:

https://drive.google.com/file/d/12Pbbfb … sp=sharing

It looks like I could just use an applescript, or even display position and click, to engage the GUI control point by clicking the GUI control radio button within red box number 1 at the bottom.  Once that is done, the application creates a second GUI control point inside the red box number 2.  If I use a mouse, I can now click, hold, and drag that second GUI control point and move it around, affecting the Curves control in DaVinci Resolve.

2.  How can instead create an applescript or command line shell to:

a.  Select the first GUI control point (box 1)
b.  Identify the second GUI control point (box 2) created after the selection of GUI control point 1.
c.  Click and hold the second GUI control point (box 2) so that I can use a Kensington trackball to move it around.
d.  Assign a button to release control of the second GUI control point.

You can see Xcode's UI browser provides a performable action for GUI point 1 (box 1):

I can't find an executable action listed in the four UI GUI inspectors for the second GUI control point (box 2), when I click on that second GUI control point:

Xcode UI browser:

https://drive.google.com/file/d/1QYA8ew … sp=sharing

The whole box is green, does that mean the whole box is treated as one selection, and nothing inside can be scripted?:

https://drive.google.com/file/d/16Cy_yF … sp=sharing

Here are two macro UI identifiers generously provided on KM's forum:

https://drive.google.com/file/d/1bC29qk … sp=sharing
https://drive.google.com/file/d/1CXE-05 … sp=sharing

Thanks!

Last edited by whitebalance (2021-01-18 05:33:55 pm)

Offline

 

#2 2021-01-18 07:48:54 pm

Fredrik71
Member
Registered: 2019-10-23
Posts: 630

Re: Script a GUI item that doesn't appear in UI browers?

If you like to make mouseclick and hold, move in x,y position you could do that with:

MouseTools: http://www.hamsoftengineering.com/codeS … Tools.html

Or

CliClick: https://www.bluem.net/jump/cliclick

its command-line, so you use do shell script or terminal.

Regards


The purpose to study someone else art is not to add, its to make less more.

Offline

 

#3 2021-01-18 10:57:35 pm

KniazidisR
Member
From:: Greece
Registered: 2019-03-03
Posts: 1645

Re: Script a GUI item that doesn't appear in UI browers?

whitebalance wrote:


a.  Select the first GUI control point (box 1)
b.  Identify the second GUI control point (box 2) created after the selection of GUI control point 1.
c.  Click and hold the second GUI control point (box 2) so that I can use a Kensington trackball to move it around.
d.  Assign a button to release control of the second GUI control point.



1) Open your interface
2) Open Automator, Choose Create Quick Action.
3) Click RED button (it is RECORD function of Automator, otherwise, its Follow Me function). It will record your clicks
4) perform in your inteface your actions  a. b. c. d. manually
5) Stop Automator Recording

Now the Automator has recorded your manual actions. In Automator window see what script wrote Automator for you smile If UI element you need exist, it should be included in the recorded Automator's script.

If it doesn't exist, then you should use mouse tools, as suggested by Fredrik71

NOTE: to see Automator's script, you should drag recorded actions to Automator's code area, one by one

Last edited by KniazidisR (2021-01-18 11:04:03 pm)


Model: MacBook Pro
OS X: Catalina 10.15.4
Web Browser: Safari 13.1
Ram: 4 GB

Offline

 

#4 2021-01-19 07:07:31 pm

whitebalance
Member
Registered: 2021-01-18
Posts: 10

Re: Script a GUI item that doesn't appear in UI browers?

Fredrik71 wrote:

If you like to make mouseclick and hold, move in x,y position you could do that with:

MouseTools: http://www.hamsoftengineering.com/codeS … Tools.html

Or

CliClick: https://www.bluem.net/jump/cliclick

its command-line, so you use do shell script or terminal.

Regards



Hello, is mouseclick able to determine the x and y position of a GUI item on it's own?  The point is that the second GUI element is movable.  Once it's moved, how do I script a way of finding it again, selecting, so that I may move it again?  It's not a fixed GUI location.  It can be anywhere.

Thanks!

Last edited by whitebalance (2021-01-19 07:07:58 pm)

Offline

 

#5 2021-01-20 03:30:35 am

Fredrik71
Member
Registered: 2019-10-23
Posts: 630

Re: Script a GUI item that doesn't appear in UI browers?

There are way to get x,y screen coordinates from a window.

1. Properties of UI element
2. Return the attributes
3. Properties of application  window

ex1.

Applescript:

tell application "Terminal" to tell front window
   set theFrame to its frame
end tell

ex2.

Applescript:

tell application "System Events" to tell process "Terminal"
   set frontmost to true
   tell front window
       set thePosition to position
   end tell
end tell

ex3.

Applescript:

tell application "System Events" to tell process "Terminal"
   set frontmost to true
   tell front window
       its value of attribute "AXFrame"
   end tell
end tell

Cliclick or MouseTools do not know anything about windows size or position.
Its could only get/set properties from mouse and input events ex. keyboard



So how did I know 'frame, position and AXFrame'... its very simple we ask for properties.

ex1.
We ask for properties in tell block of front window

Applescript:

tell application "Terminal" to tell front window
   its properties
end tell

ex2.
We ask for properties in tell block of front window

Applescript:

tell application "System Events" to tell process "Terminal"
   set frontmost to true
   tell front window
       its properties
   end tell
end tell

ex3.
We ask for:

Applescript:

its properties of attributes

When we know its AXFrame attribute, we could do.

Applescript:

its properties of attribute "AXFrame"
--> {value:{0, 511, 710, 900}, class:attribute, settable:false, name:"AXFrame"}

Now we could ask for values.

Applescript:

its value of attribute "AXFrame"

Last edited by Fredrik71 (2021-01-20 10:37:59 am)


The purpose to study someone else art is not to add, its to make less more.

Offline

 

#6 2021-01-20 09:01:20 pm

whitebalance
Member
Registered: 2021-01-18
Posts: 10

Re: Script a GUI item that doesn't appear in UI browers?

Ooooh!!! I'm gonna give this a try.

I was trying with image recognition, and got close, but I could not move the control point once it was selected, even though I was using the "click and hold" action in Keyboard Maestro:

https://drive.google.com/file/d/1bp5dZb … sp=sharing

https://drive.google.com/file/d/1Qp47pA … sp=sharing

Thanks, can't wait to try.

Offline

 

#7 2021-01-20 09:05:24 pm

whitebalance
Member
Registered: 2021-01-18
Posts: 10

Re: Script a GUI item that doesn't appear in UI browers?

Wait, just to be clear, those terminal scripts will identify the GUI control point in the red box, numbered 2, not the entire window?:

https://drive.google.com/file/d/12Pbbfb … sp=sharing

Thanks!

Last edited by whitebalance (2021-01-20 09:06:29 pm)

Offline

 

#8 2021-01-21 05:20:58 am

Fredrik71
Member
Registered: 2019-10-23
Posts: 630

Re: Script a GUI item that doesn't appear in UI browers?

It would be difficult to help you, when you do not snd any code of yours.
And if you did I may not have the same application on my computer to try your code.

GUI Scripting could be complex process to find what we want and brake in next release.

I have done simple tests in past to move beizer point to change a curve in AppleScript.
If I remember correct it was Pixelmator Pro...


The purpose to study someone else art is not to add, its to make less more.

Offline

 

#9 2021-01-22 11:43:12 pm

whitebalance
Member
Registered: 2021-01-18
Posts: 10

Re: Script a GUI item that doesn't appear in UI browers?

Thank you Fredrik 71...

Shell click makes it work!  I did some testing, and just want to confirm that it's the API that's stopping me from using Applescript to click?  I can get Applescript to click on other elements in the UI:

https://macscripter.net/viewtopic.php?p … 32#p205332

Thanks again!!!

Last edited by whitebalance (2021-01-22 11:45:23 pm)

Offline

 

Board footer

Powered by FluxBB

RSS (new topics) RSS (active topics)