Graphical user interface scripting, aka GUI scripting. To many it is the Holy Grail of AppleScript automation. For proof, one needs only to peruse the vast archives of any AppleScript discussion list or bulletin board. Even novice archaeologists of age-old threads can uncover multitudes of recurring questions like “How do I script an non-scriptable application?” or “I’m sending commands, but application so-and-so is just sitting there, or throwing errors. What gives?”
In the last operating system century known as OS 8 and 9, a system add-on called PreFab Player could provide hungry diggers of automation artifacts with full interface scripting. This meant that one small non-scriptable application in a workflow didn’t have to bring the excavation to a halt. Player allowed AppleScripters to manipulate the interface of applications which did not support our revered language. In turn, said scripter could simulate user interaction in lieu of true backend automation.
With the introduction of OS X, scripters were once again transported to an earlier ice age when they realized there would be no PreFab Player natively supported in the new operating system. As is often the question asked by any archaeologist worth his/her salt, we were left wondering “Why?”.
Enter GUI scripting. The release of OS X 10.2 brought an advance in AppleScript civilization which I’m not sure most of us were prepared for. (Unless you caught a hint or two in our report last year about upcoming AppleScript technologies. grin) Like Howard Carter discovering the tomb of Tutankhamen and opening that first interior door (except without the nasty curse), the promise of OS-level graphical user interface scripting support is just downright overwhelming. But just as overwhelming is the syntax involved for a scripter to work its dynastic magic, which is sometimes like deciphering hieroglyphics on the walls of of the pyramids.
At this point, I’d like to mention a debate which has begun between two distinct camps within the AppleScript community. One side raises serious concerns about what OS-level GUI scripting technology might mean for future support of AppleScript by application developers. The most serious of which is the suggestion that developers might choose not to add true scriptability to their products, citing that GUI scripting makes AS support unnecessary.
The other camp suggests that GUI scripting will merely serve as an excellent supplement to true AppleScript support. Ray Robertson of Scripting Matters, Inc. is one of these:
If you’re guessing PreFab is now fading out of UI scripting history, you’d be wrong. In collaboration with one of the pharaohs of the AppleScript world, Bill Cheeseman, and testing and human interface designer Scott Lawton, PreFab brings us UI Browser. Preliminary documentation describes UI Browser as such:
Our own Greg Spence elaborates:
This is key to taking full advantage of Apple’s GUI scripting implementation since it would be nearly impossible to successfully guess the applicable name of a widget in some interfaces; a gap wider than the Nile river. UI Browser seems to bridge that gap heroically.
If that doesn’t excite your AppleScript imagination, how about a screenshot or two?
UI Browser presents the AppleScript developer with a live heirarchal view of a target application’s GUI and widgets. This goes a long way toward harmonizing and unifying the information necessary to then employ the OS GUI scripting terminology to write scripts to control the application.
Clicking the popup button at the top of the main window reveals a list of all currently running processes. Choosing one reveals a brow sable hierarchy of interface elements for the chosen application.
But UI Browser doesn’t stop there.
WIth the Attributes drawer, one can manipulate the values of each of the interface’s elements. Values which are modifiable are indicated with a bullet in the “Mod” column. Modifying these values from within UI Browser causes them to be dynamically reflected in the running application.
The Notifications drawer ‘enables you to watch incoming notifications from the target application as your GUI script, manipulates the target application’.
The Actions drawer gives the user ‘an easy way to pick menu items or click buttons in the target application.’
The Keystrokes drawer ‘lets you experiment with command-key equivalents in the target application.’
I’m very excited about one particular planned feature.
On the other hand, a reviewer cannot be completely enamored with a new product without at least raising a feature request or suggesting a few refinements. With this in mind, I hope PreFab plans to supply very thorough documentation of the information presented by UI Browser. The heirarchal view of menu trees is elegant and straightforward enough, however, some of the data within the various drawers can be cryptic. As the line between Objective-C/Cocoa programming and AppleScript becomes more blurred, developers using these respective languages are sharing tools. This means scripters are coming into contact with Obj-C methods and terminology which are foreign to us. UI Browser is no exception. Bill Cheeseman says this is because UIB was written as a developer’s tool for the accessibility industry. However, he also suggests a future release of UIB will probably switch to AppleScript terminology.
Let’s hope that ‘probably’ turns more solid and sooner rather than later.
As feature requests go, I’d love to see UI Browser include Drag-and-Drop support. This might manifest in the ability to drag and drop an item within the menu hierarchy from the UIB window right into, say, a Script Editor window automatically pasting in the desired syntax. This behavior is demonstrated beautifully by Late Night Software in Script Debugger’s dictionary Explorer.
GUI scripting opens incredible new worlds for OS X scripters. UI Browser is set to make those worlds easier to explore.