Friday, November 16, 2018

#1 2018-02-27 10:06:21 pm

t.spoon
Member
From:: BFE, Massachusetts
Registered: 2013-01-13
Posts: 358

UI Elements shown in Accessibility Instructor not found by UI Script

I'm trying to UI script an app without a dictionary, and I can't get anything to work.

UI Browser doesn't report any info on the UI elements in browse mode. But in Screen Reader mode, it updates with relevant sounding info as I scroll over UI elements, but as soon as I lock the UI Browser window and try to get more info on an element, the UI Browser window just goes blank.

Accessibility Inspector seems to read things and lock just fine, but a script based on what it reports still don't work.

Here's what Accessibility Inspector shows for an element I want to click:

<AXApplication: "ICISS Scanner">
<AXWindow: "Channels">
  <AXPopUpButton: "0">
   <AXMenuItem: "Add…">

Attributes:
   AXRole:  "AXMenuItem"
   AXRoleDescription:  "menu item"
   AXParent:  "<AXPopUpButton: "0">"
   AXEnabled:  "true"
   AXPosition:  "x=186 y=324"
   AXSize:  "w=174 h=19"
   AXTitle:  "Add…"
   AXHelp:  "(null)"
   AXSelected (W):  "true"
   AXMenuItemCmdChar:  "(null)"
   AXMenuItemCmdVirtualKey:  "(null)"
   AXMenuItemCmdGlyph:  "(null)"
   AXMenuItemCmdModifiers:  "0"
   AXMenuItemMarkChar:  "(null)"
   AXMenuItemPrimaryUIElement:  "(null)"
   AXFrame:  "x=186 y=324 w=174 h=19"

Actions:
   AXCancel - cancel
   AXPress - press

Errors:
   AXParent:  UIElement is not included as a child of the parent AXPopUpButton.

Warnings:
   AXFrame:  Unexpected extra attribute

Optional Attributes:
   AXLinkedUIElements
   AXDescription
   AXChildren
   AXServesAsTitleForUIElements



so I scripted:

Applescript:


tell application "System Events"
   tell application process "ICISS Scanner"
       activate
       delay 0.5
       tell window "Channels"
           tell pop up button "0"
               click menu item "Add..."
           end tell
       end tell
   end tell
end tell

But this script returns the error:

"System Events got an error: Can’t get window 'Channels' of application process 'ICISS Scanner.'"

Any insights would be appreciated. Scripting this would save a mountain of time.

Thanks,

Tom.


Hackintosh built February, 2012 |  Mac OS Sierra
GIGABYTE GA-Z68X-UD3H-B3 | Core i5 2500k | 16 GB DDR3 | GIGABYTE Geforce 1050 TI 4GB
250 GB Samsung 850 EVO | 4 TB RAID
Dell Ultrasharp U3011 | Dell Ultrasharp 2007FPb

Offline

 

#2 2018-03-05 03:54:32 am

Yvan Koenig
Member
Registered: 2006-09-14
Posts: 3287

Re: UI Elements shown in Accessibility Instructor not found by UI Script

I don't own the app so I can't test.
You may try with :

Applescript:

tell application "System Events"
   tell application process "ICISS Scanner"
       activate
       delay 0.5
       tell window "Channels"
           tell pop up button "0"
               click it -- try to add it to reveal the menu items
               click menu item "Add..." -- Are you sure that it's 3 separate points ? May be the character ellipsis …
           end tell
       end tell
   end tell
end tell

Yvan KOENIG running High Sierra 10.13.3 in French (VALLAURIS, France) lundi 5 mars 2018 10:53:02

Offline

 

#3 2018-03-06 12:50:44 pm

t.spoon
Member
From:: BFE, Massachusetts
Registered: 2013-01-13
Posts: 358

Re: UI Elements shown in Accessibility Instructor not found by UI Script

Thanks, but same thing -

"System Events got an error: Can’t get window 'Channels' of application process 'ICISS Scanner.'"

At this point I've given up on this - I've used Accessibility Inspector, UI Browser, and Script Editor to read UI elements from this app and try to script interaction, and not a single UI widget ever responds to as single thing I try. I've given up on this and am using all emulated mouse clicks. Not elegant, but all I've got, and working pretty reliably.

- Tom.


Hackintosh built February, 2012 |  Mac OS Sierra
GIGABYTE GA-Z68X-UD3H-B3 | Core i5 2500k | 16 GB DDR3 | GIGABYTE Geforce 1050 TI 4GB
250 GB Samsung 850 EVO | 4 TB RAID
Dell Ultrasharp U3011 | Dell Ultrasharp 2007FPb

Offline

 

#4 2018-03-07 05:20:59 am

Yvan Koenig
Member
Registered: 2006-09-14
Posts: 3287

Re: UI Elements shown in Accessibility Instructor not found by UI Script

Maybe the problem is not related to the menu but to the name of the window.

What would be return by :

Applescript:

tell application "System Events"
   tell application process "ICISS Scanner"
       activate
       delay 0.5
       set frontmost to true
       return name of every window
       (*
       tell window "Channels"
           tell pop up button "0"
               click it -- try to add it to reveal the menu items
               click menu item "Add..." -- Are you sure that it's 3 separate points ? May be the character ellipsis …
           end tell
       end tell
       *)

   end tell
end tell

But in fact I'm quite sure that the error message doesn't match the real problem.
I carefully asked you if you were sure of the name of the menu item.
It appears that you didn't : in the log posted in your first message, the menu item is named : "Add…"  there aren't three dots but a single ellipsis character.

Yvan KOENIG running High Sierra 10.13.3 in French (VALLAURIS, France) mercredi 7 mars 2018 12:16:29

Offline

 

#5 2018-03-07 06:37:47 am

Nigel Garvey
Moderator
From:: Warwickshire, England
Registered: 2002-11-20
Posts: 4720

Re: UI Elements shown in Accessibility Instructor not found by UI Script

t.spoon wrote:

so I scripted:

Applescript:


tell application "System Events"
   tell application process "ICISS Scanner"
       activate


'activate' should be 'set frontmost to true' or 'set its frontmost to true'. You're addressing an application process of System Events, not an application in its own right. It may also be worth checking that the process has the same name as the application.


NG

Offline

 

#6 2018-03-07 01:56:47 pm

t.spoon
Member
From:: BFE, Massachusetts
Registered: 2013-01-13
Posts: 358

Re: UI Elements shown in Accessibility Instructor not found by UI Script

Thanks for the help. Progress!

Yvan,

After your first reply, I did check Accessibility Inspector again, found it's a "…" (single character ellipsis) not a "..." (three periods),  but correcting that didn't have any effect. I should have mentioned I had tried that in my last post, thank you for the suggestion.

Nigel,

Thanks, I had noticed that too, and tried changing it to:

Applescript:

tell application "ICISS Scanner" to activate
delay 0.2

tell application "System Events"
   tell application process "ICISS Scanner"
-- [etc]

Running this:

Applescript:

tell application "ICISS Scanner" to activate
delay 0.2

tell application "System Events"
   tell application process "ICISS Scanner"
       delay 0.2
       set theWindows to the name of every window
       set theElements to the name of every UI element of window "Channels"
   end tell
end tell

"theWindows" is:

Applescript:

{"Script", "Color Converter", "Info", "Channels", "Island Preseps.psd"}

Which sounds promising, but "theElements" is:
{"Channels", missing value, "111%", "∑:", missing value, missing value}
Which doesn't sound promising at all.
The "Channels" and "∑:" elements are static text, the "111%" is a changing readout of a value the mouse is over on an image. Everything else it sees is "missing value," and even then, 3 missing values isn't nearly enough UI elements for the window.

Although it's interesting to me that when I tell it to use a UI element, the error message is that it "Can’t get window 'Channels,'" but it actually can get the window "channels," just nothing of use below that level.

I dug deeper into the "missing value" name elements:

Applescript:

tell application "ICISS Scanner" to activate
delay 0.2

tell application "System Events"
   tell application process "ICISS Scanner"
       delay 0.2
       set channelsWindow to window "Channels"
       set namesNroles to {the name of every UI element of channelsWindow, the role of every UI element of channelsWindow}
   end tell
end tell

which revealed that UI Element 6 is a pop up button:

Applescript:

{{"Channels", missing value, "246%", "∑:", missing value, missing value}, {"AXStaticText", "AXGrowArea", "AXStaticText", "AXStaticText", "AXScrollBar", "AXPopUpButton"}}

Progress... tried this:

Applescript:

tell application "System Events"
   tell application process "ICISS Scanner"
       delay 0.2
       set channelsWindow to window "Channels"
       set flyoutMenu to the first UI element of channelsWindow whose role is "AXPopUpButton"
       tell flyoutMenu
           click it
           delay 0.1
           click menu item "Add…"
       end tell
   end tell
end tell

New error - "can't get menu item 'Add…'

I never see the menu pull out following the "click it" command. In case just that's going wrong, I tried this:

Applescript:

tell application "ICISS Scanner" to activate
delay 0.2

tell application "System Events"
   tell application process "ICISS Scanner"
       delay 2
       set channelsWindow to window "Channels"
       set flyoutMenu to the first UI element of channelsWindow whose role is "AXPopUpButton"
       tell flyoutMenu
           set menuProps to the properties
       end tell
   end tell
end tell

And then with that 2-second delay I manually clicked the pop-up menu and left it up, hoping the script could read it's contents. No luck there, menuProps is:

Applescript:

{minimum value:0, orientation:missing value, position:{183, 298}, role description:"pop up button", accessibility description:missing value, focused:false, title:"", size:{16, 16}, value:0, help:missing value, enabled:true, maximum value:1, role:"AXPopUpButton", entire contents:{}, subrole:missing value, selected:missing value, name:missing value, description:"pop up button"}

Tried this:

Applescript:

tell application "ICISS Scanner" to activate
delay 0.2

tell application "System Events"
   tell application process "ICISS Scanner"
       delay 2
       set channelsWindow to window "Channels"
       set flyoutMenu to the first UI element of channelsWindow whose role is "AXPopUpButton"
       tell flyoutMenu
           set menuElements to every UI element
       end tell
   end tell
end tell

With the manual click.

now menuElements is:

menu 1 of pop up button 1 of window "Channels" of application process "ICISS Scanner"

so tried this:

Applescript:

tell application "ICISS Scanner" to activate
delay 0.2

tell application "System Events"
   tell application process "ICISS Scanner"
       delay 2
       set channelsWindow to window "Channels"
       set flyoutMenu to the first UI element of channelsWindow whose role is "AXPopUpButton"
       tell flyoutMenu
           tell menu 1
               set menuElements to every UI element
           end tell
       end tell
   end tell
end tell

With the manual click, and menuElements is now:

Applescript:

{menu item "Edit…" of menu 1 of pop up button 1 of window "Channels" of application process "ICISS Scanner", menu item "Add…" of menu 1 of pop up button 1 of window "Channels" of application process "ICISS Scanner", menu item "Remove" of menu 1 of pop up button 1 of window "Channels" of application process "ICISS Scanner", menu item 4 of menu 1 of pop up button 1 of window "Channels" of application process "ICISS Scanner", menu item "Preferences…" of menu 1 of pop up button 1 of window "Channels" of application process "ICISS Scanner", menu item "Switch reading" of menu 1 of pop up button 1 of window "Channels" of application process "ICISS Scanner", menu item "Switch colormodel" of menu 1 of pop up button 1 of window "Channels" of application process "ICISS Scanner", menu item "Show input channels" of menu 1 of pop up button 1 of window "Channels" of application process "ICISS Scanner", menu item 9 of menu 1 of pop up button 1 of window "Channels" of application process "ICISS Scanner", menu item "Load Channelset…" of menu 1 of pop up button 1 of window "Channels" of application process "ICISS Scanner", menu item "Save Channelset…" of menu 1 of pop up button 1 of window "Channels" of application process "ICISS Scanner", menu item 12 of menu 1 of pop up button 1 of window "Channels" of application process "ICISS Scanner", menu item "Save as default…" of menu 1 of pop up button 1 of window "Channels" of application process "ICISS Scanner", menu item "Reset to default" of menu 1 of pop up button 1 of window "Channels" of application process "ICISS Scanner", menu item 15 of menu 1 of pop up button 1 of window "Channels" of application process "ICISS Scanner", menu item " Empty" of menu 1 of pop up button 1 of window "Channels" of application process "ICISS Scanner", menu item "Euroscale CMYK" of menu 1 of pop up button 1 of window "Channels" of application process "ICISS Scanner", menu item "Pure CMYK" of menu 1 of pop up button 1 of window "Channels" of application process "ICISS Scanner", menu item "RGB" of menu 1 of pop up button 1 of window "Channels" of application process "ICISS Scanner"}

So that's great.

But this does rely on the manual click - if I take it out, I get "can't get 'menu item 1."

Same with this: "can't get 'menu item 1.'"

Applescript:

tell application "ICISS Scanner" to activate
delay 0.2

tell application "System Events"
   tell application process "ICISS Scanner"
       delay 2
       set channelsWindow to window "Channels"
       set flyoutMenu to the first UI element of channelsWindow whose role is "AXPopUpButton"
       tell flyoutMenu
           click it
           delay 0.1
           tell menu 1
               click it
               delay 0.1
               set menuElements to every UI element
           end tell
       end tell
   end tell
end tell

Any ideas where the "click" could be going wrong?

I'm curious if anyone has any more ideas, but this isn't a bottleneck for me anymore because I have an inelegant but functional mouse-clicking script automating everything I need from this application.


Hackintosh built February, 2012 |  Mac OS Sierra
GIGABYTE GA-Z68X-UD3H-B3 | Core i5 2500k | 16 GB DDR3 | GIGABYTE Geforce 1050 TI 4GB
250 GB Samsung 850 EVO | 4 TB RAID
Dell Ultrasharp U3011 | Dell Ultrasharp 2007FPb

Offline

 

#7 2018-03-07 02:14:43 pm

t.spoon
Member
From:: BFE, Massachusetts
Registered: 2013-01-13
Posts: 358

Re: UI Elements shown in Accessibility Instructor not found by UI Script

This works, but is still using AS Toolbox to click, but getting the coordinates to click at much more directly through UI scripting. Let me know if anyone has any ideas regarding the UI scripting "clicks" failing. It seems like at this point, I've got everything else I'd need to get away from the mouse clicks.

Thanks again,
Tom.

Applescript:

tell application "System Events"
   tell application process "ICISS Scanner"
       set frontmost to true
       delay 0.2
       set channelsWindow to window "Channels"
       set flyoutMenu to the first UI element of channelsWindow whose role is "AXPopUpButton"
       set flyoutPosition to the position of flyoutMenu
       AST click at {(item 1 of flyoutPosition), ((item 2 of flyoutPosition) + 7)}
       tell flyoutMenu
           delay 0.4
           tell menu 1
               delay 0.2
               set addMenuPositon to the position of menu item "Add…"
               AST click at addMenuPositon
           end tell
       end tell
   end tell
end tell

Last edited by t.spoon (2018-03-07 02:15:01 pm)


Hackintosh built February, 2012 |  Mac OS Sierra
GIGABYTE GA-Z68X-UD3H-B3 | Core i5 2500k | 16 GB DDR3 | GIGABYTE Geforce 1050 TI 4GB
250 GB Samsung 850 EVO | 4 TB RAID
Dell Ultrasharp U3011 | Dell Ultrasharp 2007FPb

Offline

 

#8 2018-03-08 04:44:50 am

Yvan Koenig
Member
Registered: 2006-09-14
Posts: 3287

Re: UI Elements shown in Accessibility Instructor not found by UI Script

Try with :

Applescript:

tell application "ICISS Scanner" to activate
delay 0.2

tell application "System Events"
   tell application process "ICISS Scanner"
       set frontmost to true
       set channelsWindow to window "Channels"
       --set flyoutMenu to the first UI element of channelsWindow whose role is "AXPopUpButton"
       set flyoutMenu to pop up button 1 of channelsWindow
       log (get flyoutMenu's name) --> "∑:"
       tell flyoutMenu
           click it
           delay 0.1
           tell menu 1
               set menuElements to name of its menu items
               log menuElements --> {"Edit…" , "Add…" , "Remove" , 4 , "Preferences…" , "Switch reading" , "Switch colormodel" , "Show input channels" , 9 , "Load Channelset…" , "Save Channelset…" , 12 , "Save as default…" , "Reset to default" , 15 , " Empty" , "Euroscale CMYK" , "Pure CMYK" , "RGB"}
               click menu item 2
           end tell
       end tell
   end tell
end tell

If it doesn't do the trick it's that the application doesn't treat correctly the click command.
I found some which behave this way when they are urged to click in buttons or checkboxes, never saw that with pop ups.
If it's really the app which doesn't treat the click you must rely upon AS Toolbox or mouseClick.

Yvan KOENIG running High Sierra 10.13.3 in French (VALLAURIS, France) jeudi 8 mars 2018 11:44:44

Last edited by Yvan Koenig (2018-03-08 04:46:19 am)

Offline

 

#9 2018-03-08 02:55:36 pm

t.spoon
Member
From:: BFE, Massachusetts
Registered: 2013-01-13
Posts: 358

Re: UI Elements shown in Accessibility Instructor not found by UI Script

Applescript:

log (get flyoutMenu's name)

yields

(*missing value*)



then when it gets to

Applescript:

set menuElements to name of its menu items

it errors:

System Events got an error: Can’t get menu 1 of pop up button 1 of window "Channels" of application process "ICISS Scanner". Invalid index.



I think I'm just stuck with mouse clicking here.

But thanks for the help, everyone. I've learned how to dig deeper with UI scripting than I had in the past, I'm sure it will come in handy in the future.

Most importantly, I hadn't realized that using Script Debugger's ability to read application UI's on the info screen sometimes can't read some elements, but if you activate the program and then set that info to a variable you can get it. I guess the program has to be frontmost for the information to be accessible.

Thanks,

Tom.


Hackintosh built February, 2012 |  Mac OS Sierra
GIGABYTE GA-Z68X-UD3H-B3 | Core i5 2500k | 16 GB DDR3 | GIGABYTE Geforce 1050 TI 4GB
250 GB Samsung 850 EVO | 4 TB RAID
Dell Ultrasharp U3011 | Dell Ultrasharp 2007FPb

Offline

 

#10 2018-03-12 08:56:16 am

DJ Bazzie Wazzie
Member
From:: the Netherlands
Registered: 2004-10-20
Posts: 2808
Website

Re: UI Elements shown in Accessibility Instructor not found by UI Script

For the record AS Toolbox doesn't really perform a click but only a mouse down event with an click state which is fine for clicking in Cocoa and Carbon applications. This way it can perform an double click or triple click (in text).

Both System Events and AS Toolbox are not performing a sequence of mouse down and mouse up event but MouseTools from Hank (HAMSoft Engineering) does the job. It may be required if the application has it's own layout manager and uses different event handling for clicks than the standard OS. It could be worth the try…

You can download it here

update: Proof of concept, with empty menu's created on the fly AST does work well (bot so does System Events in this case)

Applescript:

tell application "System Events"
   tell application process "Script Editor"
       set frontmost to true
       delay 0.2
       set SEWindow to window 1 --window containing this script
       set flyoutMenu to pop up button 1 of group 1 of SEWindow
       set flyoutPosition to the position of flyoutMenu
       AST click at {(item 1 of flyoutPosition), ((item 2 of flyoutPosition) + 7)}
   end tell
end tell

Last edited by DJ Bazzie Wazzie (2018-03-12 08:58:52 am)

Offline

 

#11 2018-11-06 08:27:55 am

t.spoon
Member
From:: BFE, Massachusetts
Registered: 2013-01-13
Posts: 358

Re: UI Elements shown in Accessibility Instructor not found by UI Script

Thanks for the tip.

I tried removing "application" and just using "tell process "ICISS Scanner," but it still didn't receive the clicks.


Hackintosh built February, 2012 |  Mac OS Sierra
GIGABYTE GA-Z68X-UD3H-B3 | Core i5 2500k | 16 GB DDR3 | GIGABYTE Geforce 1050 TI 4GB
250 GB Samsung 850 EVO | 4 TB RAID
Dell Ultrasharp U3011 | Dell Ultrasharp 2007FPb

Offline

 

#12 2018-11-06 09:06:09 am

Yvan Koenig
Member
Registered: 2006-09-14
Posts: 3287

Re: UI Elements shown in Accessibility Instructor not found by UI Script

I'm not surprised by your problems with click.
Even in some Apple application I found cases where standard click doesn't respond forcing me to use the CLI name cliclick.
When I try to navigate in a GUI, I rarely work with names but with classes.
In your case I assume that some of the elements returning missing value are containers of other UI elements.
I will search in my archives for a sample code using cliclick and class.

Yvan KOENIG running High Sierra 10.13.6 in French (VALLAURIS, France) mardi 6 novembre 2018 16:06:02

Offline

 

#13 2018-11-07 03:35:31 am

CleMacS
Member
Registered: 2015-12-11
Posts: 6

Re: UI Elements shown in Accessibility Instructor not found by UI Script

Hi, I was looking myself for a way to click the Notification Button to the ultra right of my (icon) menu bar.

Now, what I post below is a way to click a non-system menu bar icon, that was NOT accessible via its AppleScript "path" (menu "NotificationCenter" of menu bar item "NotificationCenter" of menu bar 1 of process "NotificationCenter" of application "System Events" // translated roughly from German).

I have a vague hope that it might work for you, too! (I took values for "click at" from your 1st post.)>>

Applescript:


tell application "System Events" to tell application "ICISS Scanner"
    activate
    delay 0.2 -- ( << NOT sure this is necessary)
    tell process "SystemUIServer" to click at {186, 324}
end telll

Last edited by CleMacS (2018-11-07 03:37:00 am)

Offline

 

#14 2018-11-07 07:46:04 am

t.spoon
Member
From:: BFE, Massachusetts
Registered: 2013-01-13
Posts: 358

Re: UI Elements shown in Accessibility Instructor not found by UI Script

CleMacS,

Thanks, but I think you may have missed some bits of information in this long thread:

I'm curious if anyone has any more ideas, but this isn't a bottleneck for me anymore because I have an inelegant but functional mouse-clicking script automating everything I need from this application.



I've had this script used in a production environment for about 8 months now. I get the location of the UI widgets from reading the GUI elements with System Events and then perform clicks with AppleScript Toolbox, running embedded in the script.

https://astoolbox.wordpress.com

I also have to perform some click and drag operations in the UI, which I'm doing with Python.

https://macscripter.net/viewtopic.php?id=46154


Hackintosh built February, 2012 |  Mac OS Sierra
GIGABYTE GA-Z68X-UD3H-B3 | Core i5 2500k | 16 GB DDR3 | GIGABYTE Geforce 1050 TI 4GB
250 GB Samsung 850 EVO | 4 TB RAID
Dell Ultrasharp U3011 | Dell Ultrasharp 2007FPb

Offline

 

Board footer

Powered by FluxBB

RSS (new topics) RSS (active topics)