UI Elements shown in Accessibility Instructor not found by UI Script

I’m trying to UI script an app without a dictionary, and I can’t get anything to work.

UI Browser doesn’t report any info on the UI elements in browse mode. But in Screen Reader mode, it updates with relevant sounding info as I scroll over UI elements, but as soon as I lock the UI Browser window and try to get more info on an element, the UI Browser window just goes blank.

Accessibility Inspector seems to read things and lock just fine, but a script based on what it reports still don’t work.

Here’s what Accessibility Inspector shows for an element I want to click:

so I scripted:


tell application "System Events"
	tell application process "ICISS Scanner"
		activate
		delay 0.5
		tell window "Channels"
			tell pop up button "0"
				click menu item "Add..."
			end tell
		end tell
	end tell
end tell

But this script returns the error:

“System Events got an error: Can’t get window ‘Channels’ of application process ‘ICISS Scanner.’”

Any insights would be appreciated. Scripting this would save a mountain of time.

Thanks,

Tom.

I don’t own the app so I can’t test.
You may try with :

tell application "System Events"
	tell application process "ICISS Scanner"
		activate
		delay 0.5
		tell window "Channels"
			tell pop up button "0"
				click it -- try to add it to reveal the menu items
				click menu item "Add..." -- Are you sure that it's 3 separate points ? May be the character ellipsis …
			end tell
		end tell
	end tell
end tell

Yvan KOENIG running High Sierra 10.13.3 in French (VALLAURIS, France) lundi 5 mars 2018 10:53:02

Thanks, but same thing -

“System Events got an error: Can’t get window ‘Channels’ of application process ‘ICISS Scanner.’”

At this point I’ve given up on this - I’ve used Accessibility Inspector, UI Browser, and Script Editor to read UI elements from this app and try to script interaction, and not a single UI widget ever responds to as single thing I try. I’ve given up on this and am using all emulated mouse clicks. Not elegant, but all I’ve got, and working pretty reliably.

  • Tom.

Maybe the problem is not related to the menu but to the name of the window.

What would be return by :

tell application "System Events"
	tell application process "ICISS Scanner"
		activate
		delay 0.5
		set frontmost to true
		return name of every window
		(*
		tell window "Channels"
			tell pop up button "0"
				click it -- try to add it to reveal the menu items
				click menu item "Add..." -- Are you sure that it's 3 separate points ? May be the character ellipsis …
			end tell
		end tell
		*)
	end tell
end tell

But in fact I’m quite sure that the error message doesn’t match the real problem.
I carefully asked you if you were sure of the name of the menu item.
It appears that you didn’t : in the log posted in your first message, the menu item is named : “Add…” there aren’t three dots but a single ellipsis character.

Yvan KOENIG running High Sierra 10.13.3 in French (VALLAURIS, France) mercredi 7 mars 2018 12:16:29

‘activate’ should be ‘set frontmost to true’ or ‘set its frontmost to true’. You’re addressing an application process of System Events, not an application in its own right. It may also be worth checking that the process has the same name as the application.

Thanks for the help. Progress!

Yvan,

After your first reply, I did check Accessibility Inspector again, found it’s a “…” (single character ellipsis) not a “…” (three periods), but correcting that didn’t have any effect. I should have mentioned I had tried that in my last post, thank you for the suggestion.

Nigel,

Thanks, I had noticed that too, and tried changing it to:

tell application "ICISS Scanner" to activate
delay 0.2

tell application "System Events"
	tell application process "ICISS Scanner"
 -- [etc]

Running this:

tell application "ICISS Scanner" to activate
delay 0.2

tell application "System Events"
	tell application process "ICISS Scanner"
		delay 0.2
		set theWindows to the name of every window
		set theElements to the name of every UI element of window "Channels"
	end tell
end tell

“theWindows” is:

{"Script", "Color Converter", "Info", "Channels", "Island Preseps.psd"}

Which sounds promising, but “theElements” is:
{“Channels”, missing value, “111%”, “∑:”, missing value, missing value}
Which doesn’t sound promising at all.
The “Channels” and “∑:” elements are static text, the “111%” is a changing readout of a value the mouse is over on an image. Everything else it sees is “missing value,” and even then, 3 missing values isn’t nearly enough UI elements for the window.

Although it’s interesting to me that when I tell it to use a UI element, the error message is that it “Can’t get window ‘Channels,’” but it actually can get the window “channels,” just nothing of use below that level.

I dug deeper into the “missing value” name elements:

tell application "ICISS Scanner" to activate
delay 0.2

tell application "System Events"
	tell application process "ICISS Scanner"
		delay 0.2
		set channelsWindow to window "Channels"
		set namesNroles to {the name of every UI element of channelsWindow, the role of every UI element of channelsWindow}
	end tell
end tell

which revealed that UI Element 6 is a pop up button:

{{"Channels", missing value, "246%", "∑:", missing value, missing value}, {"AXStaticText", "AXGrowArea", "AXStaticText", "AXStaticText", "AXScrollBar", "AXPopUpButton"}}

Progress… tried this:

tell application "System Events"
	tell application process "ICISS Scanner"
		delay 0.2
		set channelsWindow to window "Channels"
		set flyoutMenu to the first UI element of channelsWindow whose role is "AXPopUpButton"
		tell flyoutMenu
			click it
			delay 0.1
			click menu item "Add…"
		end tell
	end tell
end tell

New error - "can’t get menu item ‘Add…’

I never see the menu pull out following the “click it” command. In case just that’s going wrong, I tried this:

tell application "ICISS Scanner" to activate
delay 0.2

tell application "System Events"
	tell application process "ICISS Scanner"
		delay 2
		set channelsWindow to window "Channels"
		set flyoutMenu to the first UI element of channelsWindow whose role is "AXPopUpButton"
		tell flyoutMenu
			set menuProps to the properties
		end tell
	end tell
end tell

And then with that 2-second delay I manually clicked the pop-up menu and left it up, hoping the script could read it’s contents. No luck there, menuProps is:

{minimum value:0, orientation:missing value, position:{183, 298}, role description:"pop up button", accessibility description:missing value, focused:false, title:"", size:{16, 16}, value:0, help:missing value, enabled:true, maximum value:1, role:"AXPopUpButton", entire contents:{}, subrole:missing value, selected:missing value, name:missing value, description:"pop up button"}

Tried this:

tell application "ICISS Scanner" to activate
delay 0.2

tell application "System Events"
	tell application process "ICISS Scanner"
		delay 2
		set channelsWindow to window "Channels"
		set flyoutMenu to the first UI element of channelsWindow whose role is "AXPopUpButton"
		tell flyoutMenu
			set menuElements to every UI element
		end tell
	end tell
end tell

With the manual click.

now menuElements is:

menu 1 of pop up button 1 of window “Channels” of application process “ICISS Scanner”

so tried this:

tell application "ICISS Scanner" to activate
delay 0.2

tell application "System Events"
	tell application process "ICISS Scanner"
		delay 2
		set channelsWindow to window "Channels"
		set flyoutMenu to the first UI element of channelsWindow whose role is "AXPopUpButton"
		tell flyoutMenu
			tell menu 1
				set menuElements to every UI element
			end tell
		end tell
	end tell
end tell

With the manual click, and menuElements is now:

{menu item "Edit…" of menu 1 of pop up button 1 of window "Channels" of application process "ICISS Scanner", menu item "Add…" of menu 1 of pop up button 1 of window "Channels" of application process "ICISS Scanner", menu item "Remove" of menu 1 of pop up button 1 of window "Channels" of application process "ICISS Scanner", menu item 4 of menu 1 of pop up button 1 of window "Channels" of application process "ICISS Scanner", menu item "Preferences…" of menu 1 of pop up button 1 of window "Channels" of application process "ICISS Scanner", menu item "Switch reading" of menu 1 of pop up button 1 of window "Channels" of application process "ICISS Scanner", menu item "Switch colormodel" of menu 1 of pop up button 1 of window "Channels" of application process "ICISS Scanner", menu item "Show input channels" of menu 1 of pop up button 1 of window "Channels" of application process "ICISS Scanner", menu item 9 of menu 1 of pop up button 1 of window "Channels" of application process "ICISS Scanner", menu item "Load Channelset…" of menu 1 of pop up button 1 of window "Channels" of application process "ICISS Scanner", menu item "Save Channelset…" of menu 1 of pop up button 1 of window "Channels" of application process "ICISS Scanner", menu item 12 of menu 1 of pop up button 1 of window "Channels" of application process "ICISS Scanner", menu item "Save as default…" of menu 1 of pop up button 1 of window "Channels" of application process "ICISS Scanner", menu item "Reset to default" of menu 1 of pop up button 1 of window "Channels" of application process "ICISS Scanner", menu item 15 of menu 1 of pop up button 1 of window "Channels" of application process "ICISS Scanner", menu item " Empty" of menu 1 of pop up button 1 of window "Channels" of application process "ICISS Scanner", menu item "Euroscale CMYK" of menu 1 of pop up button 1 of window "Channels" of application process "ICISS Scanner", menu item "Pure CMYK" of menu 1 of pop up button 1 of window "Channels" of application process "ICISS Scanner", menu item "RGB" of menu 1 of pop up button 1 of window "Channels" of application process "ICISS Scanner"}

So that’s great.

But this does rely on the manual click - if I take it out, I get “can’t get 'menu item 1.”

Same with this: “can’t get ‘menu item 1.’”

tell application "ICISS Scanner" to activate
delay 0.2

tell application "System Events"
	tell application process "ICISS Scanner"
		delay 2
		set channelsWindow to window "Channels"
		set flyoutMenu to the first UI element of channelsWindow whose role is "AXPopUpButton"
		tell flyoutMenu
			click it
			delay 0.1
			tell menu 1
				click it
				delay 0.1
				set menuElements to every UI element
			end tell
		end tell
	end tell
end tell

Any ideas where the “click” could be going wrong?

I’m curious if anyone has any more ideas, but this isn’t a bottleneck for me anymore because I have an inelegant but functional mouse-clicking script automating everything I need from this application.

This works, but is still using AS Toolbox to click, but getting the coordinates to click at much more directly through UI scripting. Let me know if anyone has any ideas regarding the UI scripting “clicks” failing. It seems like at this point, I’ve got everything else I’d need to get away from the mouse clicks.

Thanks again,
Tom.

tell application "System Events"
	tell application process "ICISS Scanner"
		set frontmost to true
		delay 0.2
		set channelsWindow to window "Channels"
		set flyoutMenu to the first UI element of channelsWindow whose role is "AXPopUpButton"
		set flyoutPosition to the position of flyoutMenu
		AST click at {(item 1 of flyoutPosition), ((item 2 of flyoutPosition) + 7)}
		tell flyoutMenu
			delay 0.4
			tell menu 1
				delay 0.2
				set addMenuPositon to the position of menu item "Add…"
				AST click at addMenuPositon
			end tell
		end tell
	end tell
end tell

Try with :

tell application "ICISS Scanner" to activate
delay 0.2

tell application "System Events"
	tell application process "ICISS Scanner"
		set frontmost to true
		set channelsWindow to window "Channels"
		--set flyoutMenu to the first UI element of channelsWindow whose role is "AXPopUpButton"
		set flyoutMenu to pop up button 1 of channelsWindow
		log (get flyoutMenu's name) --> "∑:"
		tell flyoutMenu
			click it
			delay 0.1
			tell menu 1
				set menuElements to name of its menu items
				log menuElements --> {"Edit…" , "Add…" , "Remove" , 4 , "Preferences…" , "Switch reading" , "Switch colormodel" , "Show input channels" , 9 , "Load Channelset…" , "Save Channelset…" , 12 , "Save as default…" , "Reset to default" , 15 , " Empty" , "Euroscale CMYK" , "Pure CMYK" , "RGB"}
				click menu item 2
			end tell
		end tell
	end tell
end tell

If it doesn’t do the trick it’s that the application doesn’t treat correctly the click command.
I found some which behave this way when they are urged to click in buttons or checkboxes, never saw that with pop ups.
If it’s really the app which doesn’t treat the click you must rely upon AS Toolbox or mouseClick.

Yvan KOENIG running High Sierra 10.13.3 in French (VALLAURIS, France) jeudi 8 mars 2018 11:44:44

log (get flyoutMenu's name) 

yields

then when it gets to

set menuElements to name of its menu items

it errors:

I think I’m just stuck with mouse clicking here.

But thanks for the help, everyone. I’ve learned how to dig deeper with UI scripting than I had in the past, I’m sure it will come in handy in the future.

Most importantly, I hadn’t realized that using Script Debugger’s ability to read application UI’s on the info screen sometimes can’t read some elements, but if you activate the program and then set that info to a variable you can get it. I guess the program has to be frontmost for the information to be accessible.

Thanks,

Tom.

For the record AS Toolbox doesn’t really perform a click but only a mouse down event with an click state which is fine for clicking in Cocoa and Carbon applications. This way it can perform an double click or triple click (in text).

Both System Events and AS Toolbox are not performing a sequence of mouse down and mouse up event but MouseTools from Hank (HAMSoft Engineering) does the job. It may be required if the application has it’s own layout manager and uses different event handling for clicks than the standard OS. It could be worth the try…

You can download it here

update: Proof of concept, with empty menu’s created on the fly AST does work well (bot so does System Events in this case)

tell application "System Events"
	tell application process "Script Editor"
		set frontmost to true
		delay 0.2
		set SEWindow to window 1 --window containing this script
		set flyoutMenu to pop up button 1 of group 1 of SEWindow
		set flyoutPosition to the position of flyoutMenu
		AST click at {(item 1 of flyoutPosition), ((item 2 of flyoutPosition) + 7)}
	end tell
end tell

Thanks for the tip.

I tried removing “application” and just using "tell process “ICISS Scanner,” but it still didn’t receive the clicks.

I’m not surprised by your problems with click.
Even in some Apple application I found cases where standard click doesn’t respond forcing me to use the CLI name cliclick.
When I try to navigate in a GUI, I rarely work with names but with classes.
In your case I assume that some of the elements returning missing value are containers of other UI elements.
I will search in my archives for a sample code using cliclick and class.

Yvan KOENIG running High Sierra 10.13.6 in French (VALLAURIS, France) mardi 6 novembre 2018 16:06:02

Hi, I was looking myself for a way to click the Notification Button to the ultra right of my (icon) menu bar.

Now, what I post below is a way to click a non-system menu bar icon, that was NOT accessible via its AppleScript “path” (menu “NotificationCenter” of menu bar item “NotificationCenter” of menu bar 1 of process “NotificationCenter” of application “System Events” // translated roughly from German).

I have a vague hope that it might work for you, too! (I took values for “click at” from your 1st post.)>>


tell application "System Events" to tell application "ICISS Scanner"  
      activate
 &nbsp; &nbsp;  delay 0.2    -- ( << NOT sure this is necessary)
 &nbsp; &nbsp;  tell process "SystemUIServer" to click at {186, 324}
end telll

CleMacS,

Thanks, but I think you may have missed some bits of information in this long thread:

I’ve had this script used in a production environment for about 8 months now. I get the location of the UI widgets from reading the GUI elements with System Events and then perform clicks with AppleScript Toolbox, running embedded in the script.

https://astoolbox.wordpress.com

I also have to perform some click and drag operations in the UI, which I’m doing with Python.

https://macscripter.net/viewtopic.php?id=46154