Coerce GUI scripting information into string?

Over at MacOSXHints, a user named squalene contributed an Applescript that returns the names of all GUI Scripting objects in a window. Here’s the link: http://hints.macworld.com/article.php?story=20111208191312748.

And here’s squalene’s script:

-- Entire Contents Demo - mini
-- BP ages ago or so

-- This'll get all the controls and structures associated with an App's window and menus
-- In a form which is easily pasteable into your own scripts
-- and show them in the result pane below.
--
-- Copy that into a text editor and change commas to returns to get an easily  readable list.
--
-- The script can take a long time if there are LOTS of window items, such as
-- in the "music" pane of iTunes. It may even time out if you have a huge iTunes library
-- The script'll process most App's UI structures in under a minute

set appname to "System Preferences" -------------------------- Set this to the App you want to look at

set winstuff to "defaultval"
set menustuff to "defaultval"

tell application appname
  activate
end tell

tell application "System Events"
  tell process appname
    set winstuff to entire contents of front window
    set menustuff to entire contents of menu bar 1
  end tell
end tell
--return winstuff & "rrrr" & menustuff -- comment this out to get just winstuff
return winstuff -- comment this out too to get just menustuff
--return menustuff 

Question: It seems to me that it should be easy to coerce the result into a text string, with each item separated by a newline or return, but, after an hour or so of experimenting, I can’t figure out how to do it. I thought I would repeat for each item in winstuff, then add each item as a string to a variable, but I can’t coerce each item into a string. Can anyone suggest the way to do this? I think it’s easy but I just don’t know enough AppleScript to do it.

I think this can help you get started right?

set UIElements to {}
tell application "System Events"
	tell process "Finder"
		repeat with UIElement in UI elements of front window
			set end of UIElements to (role of UIElement & tab & name of UIElement as string)
		end repeat
		repeat with UIElement in UI elements of menu bar 1
			set end of UIElements to (role of UIElement & tab & name of UIElement as string)
		end repeat
	end tell
end tell

set AppleScript's text item delimiters to linefeed
set UIElements to UIElements as string
set AppleScript's text item delimiters to ""
return UIElements


That is definitely a start. Thank you. I’m going to have to work on it, because I want to use it to get information to use in a script that uses GUI scripting. The script in the first post returns a list that contains items that look like this:

I’m trying to figure out a way to get each of these items into a list (without the concluding comma).

I think you’ve given me a way to start, but I wonder if there is a way to convert the output from the original script into a set of paragraphs.

I used this additional code for a while (“targettexteditor = BBEdit”):

-----
-----
on FancyExtractThatWillProbablyFailWithTheNExtMajorOSUpdate(itemlst, targettexteditor)
	tell application "AppleScript Editor" -- First, let's get the name of our own window (yeah, we'll probably die if the 'Find' window is frontmost)
		activate
		set winam to name of front window
	end tell
	
	tell application "System Events" -- Now let's ensure the Replies pane of the Event log is visible. Instead of mouse clicks, We'll use Actions we found using "Accessability Inspector".
		tell process "AppleScript Editor"
			-- Let's use Actions we found using the UIElementInspector (Accessibility Inspector) to try to ensure the "Event Log" pane is visible.
			-- We could use "click" for this, but the AXactions are more specific.
			
			-- This little bit of GUI scripting is dodgy because the "radio group 1" buttons actually have 4 states (+/- Selected, +/- Pane visibility), but only report on 'Selected'
			-- If you hide the event log pane by clicking a radio group 1 button before running this script, the script will give an error.
			-- There's no easy workaround, but fortunately the description/event log pane is visible by default.
			-- So an actual error should be rare.
			-- When Apple comes out with a New Improved version Script Editor, this'll need to be redone.	
			if (value of radio button "Event Log" of radio group 1 of window winam) is equal to 0 then
				tell radio button "Event Log" of radio group 1 of window winam to perform action "AXPress"
				-- click radio button "Event Log" of radio group 1 of window winam -- This'll work here too
			end if
			if (value of checkbox "Replies" of group 1 of splitter group 1 of window winam) is equal to 0 then
				tell checkbox "Replies" of group 1 of splitter group 1 of window winam to perform action "AXPress"
			end if
			set rawtext to value of text area 1 of scroll area 1 of group 1 of group 1 of splitter group 1 of window winam
		end tell
	end tell
	-------------------------------------------------------------------------- The list of items that we want is now in rawtext
	-------------------------------------------------------------------------- It needs cleanup:
	set text item delimiters to {"{", "}"} -- Strip off junk from beginning and end. The stuff I want is curly bracketed.  
	set tlst to every text item of rawtext
	set text item delimiters to ""
	set rawtext to item 2 of tlst as string
	
	set text item delimiters to "," -- individual items are comma delimited. Let's change those to returns.
	set tlst to every text item of rawtext
	set text item delimiters to "\r"
	set rawtext to every text item of tlst as string
	set text item delimiters to ""
	
	tell application targettexteditor -- TextEdit,TextWrangler, BBEdit  -- Everyone has TextEdit
		activate
		--if (front document exists) is equal to false then   -- commenting this out avoids potentially unpleasant overwrites
		make new document
		--end if
		
		set text of front document to rawtext -- This'll work with "TextEdit", "TextWrangler" and "BBEdit", probably others
		--set selection of front window to rawtext -- works with TextWrangler, BBEdit, NOT TextEdit
	end tell
	
end FancyExtractThatWillProbablyFailWithTheNExtMajorOSUpdate
-----
-----

It worked when I tried it tonight.
I think it makes the code too long and complicated to post over at MacOSXHints.
Plus, it will object to updates to the targettexteditor.
I’ve not tried it under Lion with the new TextEdit for example.

Given a once in a while usage pattern for the script, I find it less hassle to do a few manual search and replace actions in my text editor of choice than to maintain a complicated handler tacked on to the end of what’s really a simple little script.

Incidentally, I’ve posted the code here too:
http://macscripter.net/viewtopic.php?pid=146486

If you’re running the script in a script editor, there’s the old “error message” hack:

property appName : "Finder" -- Your app name here.

on listToText(entireContents) -- (Handler specialised for lists of System Events references.)
	try
		|| of entireContents -- Deliberate error.
	on error stuff -- Get the error message
	end try
	
	-- Parse the message.
	set astid to AppleScript's text item delimiters
	set AppleScript's text item delimiters to {"{", "}"} -- Snow Leopard or later.
	set stuff to text from text item 2 to text item -2 of stuff
	set AppleScript's text item delimiters to "\"System Events\", "
	set stuff to stuff's text items
	set AppleScript's text item delimiters to "\"System Events\"" & linefeed
	set stuff to stuff as text
	set AppleScript's text item delimiters to astid
	
	return stuff
end listToText

on main()
	tell application "System Events"
		tell application process appName
			set frontmost to true
			set {windowExists, menuExists} to {front window exists, menu bar 1 exists}
			set {winstuff, menustuff} to {missing value, missing value}
			if (windowExists) then set winstuff to my listToText(entire contents of front window)
			if (menuExists) then set menustuff to my listToText(entire contents of menu bar 1)
		end tell
	end tell
	
	return {winstuff:winstuff, menustuff:menustuff}
end main

main()

Thank you for both replies - they both work beautifully to get the job done.

Nigel, I’m especially grateful to your code because it illustrated techniques that I was struggling to figure out on my own. For example, I spent an hour experimenting - and failing - to do what you showed how to do with the code that sets a list to {missing value, missing value}.

I’ve adapted NIgel Garvey’s script so that (1) you select the application that you want to test and (2) the results are sent to TextEdit. Apologies to Nigel if I’ve messed up his excellent work:

on run {}
	-- begin lines of simple-minded code, NOT by Nigel Garvey
	set appAlias to choose file with prompt "Select an application:"
	set appName to name of (info for appAlias)
	set appName to (the reverse of every character in appName) as string
	if ((text 1 thru 4 of appName) as string) is not equal to "ppa." then
		tell me to activate
		display dialog "Please select an application only." buttons {"OK"}
		error number -128
	end if
	set appName to (text 5 thru -1 of appName)
	set appName to (the reverse of every character in appName) as string
	set appPosix to quoted form of POSIX path of appAlias
	do shell script "open " & appPosix
	tell me to activate
	display dialog "Display the window or tab in " & appName & " that you want to analyze and click OK." buttons {"OK", "Cancel"} default button 1
	if button returned of result is "Cancel" then
		error number -128
	end if
	-- end lines NOT by Nigel Garvey
	
        --intelligent code, taken from Nigel Garvey
	tell application "System Events"
		tell application process appName
			set frontmost to true
			set {windowExists, menuExists} to {front window exists, menu bar 1 exists}
			set {winstuff, menustuff} to {missing value, missing value}
			if (windowExists) then set winstuff to my listToText(entire contents of front window)
			if (menuExists) then set menustuff to my listToText(entire contents of menu bar 1)
		end tell
	end tell
	
	-- the following block of simple-minded code is NOT by Nigel Garvey
	tell application "TextEdit"
		activate
		make new document at the front
		set the text of the front document to winstuff & return & "-----" & return & menustuff
	end tell
	-- 	return {winstuff:winstuff, menustuff:menustuff}
end run

-- the heart of the script, the intelligent stuff, by Nigel Garvey
on listToText(entireContents) -- (Handler specialised for lists of System Events references.)
	try
		|| of entireContents -- Deliberate error.
	on error stuff -- Get the error message
	end try
	
	-- Parse the message.
	set astid to AppleScript's text item delimiters
	set AppleScript's text item delimiters to {"{", "}"} -- Snow Leopard or later.
	set stuff to text from text item 2 to text item -2 of stuff
	set AppleScript's text item delimiters to "\"System Events\", "
	set stuff to stuff's text items
	set AppleScript's text item delimiters to "\"System Events\"" & linefeed
	set stuff to stuff as text
	set AppleScript's text item delimiters to astid
	
	return stuff
end listToText

Hi, emendelson.

Thanks for posting your finished script. If you’re interested, this section.

set appName to (the reverse of every character in appName) as string
if ((text 1 thru 4 of appName) as string) is not equal to "ppa." then
	tell me to activate
	display dialog "Please select an application only." buttons {"OK"}
	error number -128
end if
set appName to (text 5 thru -1 of appName)
set appName to (the reverse of every character in appName) as string

. could be simplified to this:

if (appName does not end with ".app") then
	tell me to activate
	display dialog "Please select an application only." buttons {"OK"}
	error number -128
end if
set appName to (text 1 thru -5 of appName)

Those are sweet front and back ends there.
Getting GUI info is starting to look civilized.

Thanks for the useful additions Nigel and emendelson.

or still simpler :wink:


set appAlias to choose file of type "com.apple.application-bundle" with prompt "Select an application:"
tell application "System Events" to set {name:fileName} to appAlias
set appName to text 1 thru -5 of fileName
do shell script "open " & quoted form of POSIX path of appAlias
tell me to activate
display dialog "Display the window or tab in " & appName & " that you want to analyze and click OK." buttons {"OK", "Cancel"} default button 1
-- pressing Cancel aborts the script anyway


Now that Nigel and StefanK have improved it, here’s what seems to be working code:

on run {}
	set appAlias to choose file of type "com.apple.application-bundle" with prompt "Select an application:"
	tell application "System Events" to set {name:fileName} to appAlias
	set appName to text 1 thru -5 of fileName
	do shell script "open " & quoted form of POSIX path of appAlias
	tell me to activate
	display dialog "Display the window or tab in " & appName & " that you want to analyze and click OK." buttons {"OK", "Cancel"} default button 1
	
	tell application "System Events"
		tell application process appName
			set frontmost to true
			set {windowExists, menuExists} to {front window exists, menu bar 1 exists}
			set {winstuff, menustuff} to {missing value, missing value}
			if (windowExists) then set winstuff to my listToText(entire contents of front window)
			if (menuExists) then set menustuff to my listToText(entire contents of menu bar 1)
		end tell
	end tell
	
	tell application "TextEdit"
		activate
		make new document at the front
		set the text of the front document to winstuff & return & "-----" & return & menustuff
	end tell
	-- 	return {winstuff:winstuff, menustuff:menustuff}
end run

on listToText(entireContents) -- (Handler specialised for lists of System Events references.)
	try
		|| of entireContents -- Deliberate error.
	on error stuff -- Get the error message
	end try
	
	-- Parse the message.
	set astid to AppleScript's text item delimiters
	set AppleScript's text item delimiters to {"{", "}"} -- Snow Leopard or later.
	set stuff to text from text item 2 to text item -2 of stuff
	set AppleScript's text item delimiters to "\"System Events\", "
	set stuff to stuff's text items
	set AppleScript's text item delimiters to "\"System Events\"" & linefeed
	set stuff to stuff as text
	set AppleScript's text item delimiters to astid
	
	return stuff
end listToText

Now, what I don’t understand is this: When I run this from the AppleScript Editor, I get usable human-readable code in TextEdit. But when I try to run the same script as an application, the resulting text has <> code instead of “button 1” etc. Is there a way to make the script work correctly as an application?

I took out the comments from the code that identified the author, because now ALL the intelligent code is by Nigel and StefanK, and only the trivial stuff is by me.

I looked hard, and never found one. As I mentioned over here http://hints.macworld.com/article.php?story=20111208191312748
Apple’s programmers wrote a nice parser to generate the text in the first place, but its output was never made available externally. Now you could write an app that’d send parameters to a ‘get entire contents’ script running under the Script editor, and grab the output from that, but that’d probably end up being a horrible monster to maintain.

This version has extra code in the listToText() handler which allows it to work independently of a script editor.

on run {}
	set appAlias to choose file of type "com.apple.application-bundle" with prompt "Select an application:"
	tell application "System Events" to set {name:fileName} to appAlias
	set appName to text 1 thru -5 of fileName
	do shell script "open " & quoted form of POSIX path of appAlias
	tell me to activate
	display dialog "Display the window or tab in " & appName & " that you want to analyze and click OK." buttons {"OK", "Cancel"} default button 1
	
	tell application "System Events"
		tell application process appName
			set frontmost to true
			set {windowExists, menuExists} to {front window exists, menu bar 1 exists}
			set {winstuff, menustuff} to {missing value, missing value}
			if (windowExists) then set winstuff to my listToText(entire contents of front window)
			if (menuExists) then set menustuff to my listToText(entire contents of menu bar 1)
		end tell
	end tell
	
	tell application "TextEdit"
		activate
		make new document at the front
		set the text of the front document to winstuff & return & "-----" & return & menustuff
	end tell
	-- 	return {winstuff:winstuff, menustuff:menustuff}
end run

on listToText(entireContents) -- (Handler specialised for lists of System Events references.)
	try
		|| of entireContents -- Deliberate error.
	on error stuff -- Get the error message
	end try
	
	-- Parse the message.
	set astid to AppleScript's text item delimiters
	set AppleScript's text item delimiters to {"{", "}"} -- Snow Leopard or later.
	set stuff to text from text item 2 to text item -2 of stuff
	
	-- If the list text isn't in decompiled form, create a script with the list in its source code, store it in a temporary file, and run "osadecompile" on the file.
	if (stuff does not contain "application process \"") then
		try
			set scpt to (run script "script
	tell app \"System Events\"
	{" & stuff & "}
	end
	end")
		on error errMsg
			set AppleScript's text item delimiters to astid
			tell application (path to frontmost application as text) to display dialog errMsg buttons {"OK"} default button 1 with icon caution
			return errMsg
		end try
		set tmpPath to (path to temporary items as text) & "Entire contents.scpt"
		store script scpt in file tmpPath replacing yes
		set stuff to (do shell script "osadecompile " & quoted form of POSIX path of tmpPath)
		set stuff to text from text item 2 to text item -2 of stuff
	end if
	
	set AppleScript's text item delimiters to "\"System Events\", "
	set stuff to stuff's text items
	set AppleScript's text item delimiters to "\"System Events\"" & linefeed
	set stuff to stuff as text
	set AppleScript's text item delimiters to astid
	
	return stuff
end listToText

Edit: There’s an upper limit to how much data the script can handle. ‘run script’ can only take so much text and the same appears to be true for the alternative, "osacompile’, even when the input’s a file. Even AppleScript Editor seems to seize up when running the script with this page in Safari.
Edit 2: I’ve made a cleaner exit for when ‘run script’ hits an error. The error message is now returned as the result of the listToText() handler and will appear in what’s passed to TextEdit.

Nigel,

You are indeed a virtuoso scripter! Thank you for this extraordinary code.

EDIT: And thank you for introducing me (and, I expect, many others) to the osadecompile command. Every time I ask even a question on this forum, I end up knowing far more about AppleScript than I did when I started.

I had to introduce myself to it! :wink: I knew of its existence, but had never had cause to use it before.

By the way, I’ve made a small edit in the script since you posted.

Great stuff, Nigel – I’ve used it already :smiley:

OK. Before knocking this subject on the head altogether, I’d like to offer my “director’s cut” version of the script. Assuming that the user’s most likely to have the target application open already, it asks him/her to choose from a list of running applications rather than to navigate to a file. The text output to TextEdit contains tabbed object specifiers rather than full references, which gives a clearer idea of the structure. For people like me who have TextEdit set up to default to “Wrap to Page”, the script makes the new window “Wrap to Window”. I prefer an ordinary handler to an explicit run handler. It keeps all the variables local and thus non-persistent.

-- By squaline (alias partron22?), emendelson, and Nigel Garvey.

on main()
	tell application "System Events" to set appNames to name of application processes whose visible is true
	
	tell application (path to frontmost application as text)
		set appChoice to (choose from list appNames with prompt "Which running application?" with title "Choose an application")
		if (appChoice is false) then error number -128
		set appName to item 1 of appChoice
		
		set setChoice to (choose from list {"Front Window", "Menu Bar"} with prompt "Which UI element set(s)?" with title "Choose an element set" with multiple selections allowed)
		if (setChoice is false) then error number -128
		set setFlag to ((setChoice contains "Front Window") as integer) + (((setChoice contains "Menu Bar") as integer) * 2)
	end tell
	
	tell application "System Events"
		tell application process appName
			set frontmost to true
			set {windowExists, menuExists} to {front window exists, menu bar 1 exists}
		end tell
	end tell
	set {winstuff, menustuff} to {"(No window open)", "(No menu!)"}
	if ((setFlag is not 2) and (windowExists)) then set winstuff to elementListing(appName, "front window")
	if ((setFlag > 1) and (menuExists)) then set menustuff to elementListing(appName, "menu bar 1")
	
	tell application "TextEdit"
		activate
		make new document at the front
		set the text of the front document to item setFlag of {winstuff, menustuff, winstuff & linefeed & "-----" & linefeed & menustuff}
		set WrapToWindow to text 2 thru -1 of (localized string "&Wrap to Window")
	end tell
	
	tell application "System Events"
		tell application process "TextEdit"
			tell menu item WrapToWindow of menu 1 of menu bar item 5 of menu bar 1
				if ((it exists) and (it is enabled)) then perform action "AXPress"
			end tell
		end tell
	end tell
	
	return -- nothing.
end main

on elementListing(appName, elementSet)
	-- Get a textual representation of the entire contents of the element set
	set stuff to text 2 thru -2 of (do shell script "osascript -e 'tell app \"System Events\" to tell application process \"" & appName & "\" to get entire contents of " & elementSet & "' -s s")
	
	-- If the representation contains chevron codes instead of keywords, create a script containing the representation in its source code, store it in the Temporary Items folder, and run "osadecompile" on it. (This may not be necessary now the representation's obtained with "osascript" instead of the error message hack.)
	if (stuff does not contain "application process \"") then
		try
			set scpt to (run script "script
	tell app \"System Events\"
	{" & stuff & "}
	end
	end")
		on error errMsg
			tell application (path to frontmost application as text) to display dialog errMsg buttons {"OK"} default button 1 with icon caution
			return errMsg
		end try
		set tmpPath to (path to temporary items as text) & "Entire contents.scpt"
		store script scpt in file tmpPath replacing yes
		set stuff to (do shell script "osadecompile " & quoted form of POSIX path of tmpPath)
		set stuff to text from text item 2 to text item -2 of stuff
	end if
	
	-- Break up the text, using "\"System Events\", " as a delimiter.
	set astid to AppleScript's text item delimiters
	set AppleScript's text item delimiters to "\"System Events\", "
	set stuff to stuff's text items
	-- Insert a textual "reference" to the root object at the beginning of the resulting list.
	set AppleScript's text item delimiters to " of "
	set beginning of stuff to (text from text item 2 to -1 of item 1 of stuff) & "\"System Events\""
	-- Reduce the remaining "reference" fragments to object _specifiers, tabbed according to the number of elements in the references.
	set tabs to tab & tab & tab & tab & tab & tab & tab & tab & tab & tab & tab & tab
	set tabs to tabs & tabs
	repeat with i from 2 to (count stuff)
		set thisLine to item i of stuff
		set lineBits to thisLine's text items
		-- Make sure any " of "s in element names aren't mistaken for those of the reference!
		set elementCount to 0
		set nameContinuation to false
		repeat with j from 1 to (count lineBits)
			set thisBit to item j of lineBits
			if ((not ((nameContinuation) or (thisBit contains "\""))) or ((thisBit ends with "\"") and (thisBit does not end with "\\\"")) or (thisBit ends with "\\\\\"")) then
				-- thisBit is either a complete nameless-element _specifier or it ends the right way to be either a complete named one or the completion of a name.
				set nameContinuation to false
				set elementCount to elementCount + 1
				if (elementCount is 1) then set _specifier to text 1 thru text item j of thisLine
			else
				-- The next "bit" will be the continuation of a name containing " of ".
				set nameContinuation to true
			end if
		end repeat
		set item i of stuff to (text 1 thru (elementCount - 3) of tabs) & _specifier
	end repeat
	-- Coerce back to a single text, inserting line feeds between the items.
	set AppleScript's text item delimiters to linefeed
	set stuff to stuff as text
	set AppleScript's text item delimiters to astid
	
	return stuff
end elementListing

main()

Edit: A couple of minor changes in the light of the comments below.
Edit 2: Text representations of the UI element lists now obtained using “osascript” with the -s option instead of the error-message hack. Subsidiary handler renamed to reflect the modified script layout. Length of tab string doubled. McUsr’s choice-of-element-set suggestion incorporated.
Edit 3: ‘linefeed’ instead of ‘return’ in the TextEdit code, to match the linefeeds inserted by the other handler. It makes a difference if you decide to use TextWrangler instead of TextEdit for the display.
Edit 4: Here’s an alternative elementListing() handler which uses “sed” instead of vanilla AppleScript to produce the indented text. It’s very much shorter, though not very much faster. The line endings in the shell script should all be linefeeds.

on elementListing(appName, elementSet)
	return text 1 thru -2 of (do shell script ("osascript -e 'tell app \"System Events\" to tell application process \"" & appName & "\" to get entire contents of " & elementSet & "' -s s |
sed -E 's/\\\"System Events\\\", /\\\"System Events\\\"\\'$'\\n''/g ; # Put each ''list item'' on its own line.' |
sed -E '# Make two lines from the first: the first with the root reference and the second with the indented first child specifier.
	1 s/^{(.+) of ((window |menu bar [^i]).+)$/\\2\\'$'\\n'$'\\t''\\1/ ;
	1 !{ # Extract and indent the specifiers from the remaining lines:
	s/ of ((window |menu bar [^i]).+$)?/'$'\\t''/g ; # Substitute tabs for the root reference and all '' of ''s.
	:tillDone
		s/(\\\"[^'$'\\t''\\\"]+)'$'\\t''([^\\\"]*)/\\1 of \\2/g ; # Reinstate '' of ''s between quotes.
		t tillDone
	s/'$'\\t''[^'$'\\t'']+/'$'\\t''/g ; # Zap the stuff after each tab, leaving the current specifier & trailing tabs.
	s/^([^'$'\\t'']+)(['$'\\t'']+)$/\\2\\1/ ; # Swap round the specifier and the tabs.
	} ;'") without altering line endings)
end elementListing

Nigel,

That is really superb. The tabbed output is beautifully clear. The whole script is a model of intelligence and efficiency. Thank you again.

For reasons I haven’t sorted, the final script compiles beautifully in the Script Editor (under Lion), but will not compile in Script Debugger 4.5 giving several “access not allowed” errors. It will not compile in SD4.5 in Snow Leopard either, btw, so I don’t think it’s Lion; it’s SD4.5 that’s baulking. In either case when I run it in the Script Editor is says “s” is undefined

Hi, Adam.

There is no variable ‘s’, so maybe one of the apostrophe-s-es has come adrift. Just guessing, but do you have an OSAX or something with the keyword ‘stuff’? I seem to remember having one on my old 4400 which used the StuffIt engine.

On the other hand, the ‘stuff’ variable is in the previous version of the script and didn’t cause you any trouble. :confused: