Wednesday, November 26, 2014

#1 2003-01-06 04:42:22 am

T.J. Mahaffey
Administrator
From: Arkansas, USA
Registered: 2002-11-20
Posts: 238
Website

UI Browser preview & MacOS X GUI Scripting

Graphical user interface scripting, aka GUI scripting. To many it is the Holy Grail of AppleScript automation. For proof, one needs only to peruse the vast archives of any AppleScript discussion list or bulletin board. Even novice archaeologists of age-old threads can uncover multitudes of recurring questions like "How do I script an non-scriptable application?" or "I'm sending commands, but application so-and-so is just sitting there, or throwing errors. What gives?"

In the last operating system century known as OS 8 and 9, a system add-on called PreFab Player could provide hungry diggers of automation artifacts with full interface scripting. This meant that one small non-scriptable application in a workflow didn't have to bring the excavation to a halt. Player allowed AppleScripters to manipulate the interface of applications which did not support our revered language. In turn, said scripter could simulate user interaction in lieu of true backend automation.

With the introduction of OS X, scripters were once again transported to an earlier ice age when they realized there would be no PreFab Player natively supported in the new operating system. As is often the question asked by any archaeologist worth his/her salt, we were left wondering "Why?".

Enter GUI scripting. The release of OS X 10.2 brought an advance in AppleScript civilization which I'm not sure most of us were prepared for. (Unless you caught a hint or two in our report last year about upcoming AppleScript technologies. *grin*) Like Howard Carter discovering the tomb of Tutankhamen and opening that first interior door (except without the nasty curse), the promise of OS-level graphical user interface scripting support is just downright overwhelming. But just as overwhelming is the syntax involved for a scripter to work its dynastic magic, which is sometimes like deciphering hieroglyphics on the walls of of the pyramids.

At this point, I'd like to mention a debate which has begun between two distinct camps within the AppleScript community. One side raises serious concerns about what OS-level GUI scripting technology might mean for future support of AppleScript by application developers. The most serious of which is the suggestion that developers might choose not to add true scriptability to their products, citing that GUI scripting makes AS support unnecessary.

The other camp suggests that GUI scripting will merely serve as an excellent supplement to true AppleScript support. Ray Robertson of Scripting Matters, Inc. is one of these:

"For those developers who have scriptability low on their priority lists, how much lower can it get? On the other end, having GUI scripting available is not going to affect developers like Adobe who have made scriptability a priority."

"In the meantime, the benefits of GUI scripting far outweigh any worries. Acrobat scripting is still pitiful, and yet being able to deal with the UI now makes it possible to include more Acrobat scripting as part of an automated workflow. Who knows if Quark 6 scripting will continue to decline (likely, in my opinion), and GUI scripting will help some users there."

"But beyond making non-scriptable apps scriptable or filling in holes in the implementations of other apps, GUI scripting is still needed for even the most powerful scripting implementations. I've let Apple know that my largest customers, who have scripts running on dedicated Macs 24/7, simply could not consider upgrading to X unless they have a way of dismissing unwanted dialogs. And even the most sophisticated applications throw up a dialog every now and then which you can't trap. Without Player on X--or the system-intrusive QuicKeys --there were no options."

"I'm thrilled to see GUI scripting in beta, a think it will provide a new reason for my prepress clients to move to X more quickly. But as for non-scriptable apps, there will often be plenty of essential objects we can't get at through the UI, so we can continue to lobby developers for complete scripting support."

If you're guessing PreFab is now fading out of UI scripting history, you'd be wrong. In collaboration with one of the pharaohs of the AppleScript world, Bill Cheeseman, and testing and human interface designer Scott Lawton, PreFab brings us UI Browser. Preliminary documentation describes UI Browser as such:

"UI Browser is a utility to assist AppleScripters in designing and writing scripts using Apple's new "GUI Scripting" technology. Apple introduced GUI Scripting in December 2002 for beta testing by developers and scripters. The final release of GUI Scripting will appear in a future release of Mac OS X. UI Browser works well as a tool to assist with the beta version of GUI Scripting now, and UI Browser will be updated to keep pace with Apple's GUI Scripting technology as it matures."

Our own Greg Spence elaborates:

"UI Browser has the ability to help you locate and identify Tab Sets, Sliders, Popups, Radio buttons and just about every element of a window including 'Static text'. It took me a few tries to get used to the vast amount of information it shows, but you'll quickly learn to appreciate it's value in assisting you with Apple's new 'GUI Scripting' technology."

"UI Browser takes the guess work out of trying to figure out how to identify a particular element of a window by allowing you to perform actions on the elements you have selected from the UI Browser application. I found it especially helpful with scripting the 'System Preferences' panes."

This is key to taking full advantage of Apple's GUI scripting implementation since it would be nearly impossible to successfully guess the applicable name of a widget in some interfaces; a gap wider than the Nile river. UI Browser seems to bridge that gap heroically.

If that doesn't excite your AppleScript imagination, how about a screenshot or two?

http://files.macscripter.net/unscripted/uib_browser1.jpg


UI Browser presents the AppleScript developer with a live heirarchal view of a target application's GUI and widgets. This goes a long way toward harmonizing and unifying the information necessary to then employ the OS GUI scripting terminology to write scripts to control the application.

Clicking the popup button at the top of the main window reveals a list of all currently running processes. Choosing one reveals a brow sable hierarchy of interface elements for the chosen application.

But UI Browser doesn't stop there.

http://files.macscripter.net/unscripted/uib_attributes_drawer.jpg


WIth the Attributes drawer, one can manipulate the values of each of the interface's elements. Values which are modifiable are indicated with a bullet in the "Mod" column. Modifying these values from within UI Browser causes them to be dynamically reflected in the running application.

http://files.macscripter.net/unscripted/uib_notifications_drawer.jpg


The Notifications drawer 'enables you to watch incoming notifications from the target application as your GUI script, manipulates the target application'.

http://files.macscripter.net/unscripted/uib_actions_drawer.jpg


The Actions drawer gives the user 'an easy way to pick menu items or click buttons in the target application.'

http://files.macscripter.net/unscripted/uib_keystrokes_drawer.jpg


The Keystrokes drawer 'lets you experiment with command-key equivalents in the target application.'

I'm very excited about one particular planned feature.

"In a future release, UI Browser will add the ability to 'read the screen.' You will be able to move the mouse over a widget in the target application's GUI, and UI Browser will select it for you in its main browser window, giving you immediate access to its place in the containment tree, its attributes, and so on."

On the other hand, a reviewer cannot be completely enamored with a new product without at least raising a feature request or suggesting a few refinements. With this in mind, I hope PreFab plans to supply very thorough documentation of the information presented by UI Browser. The heirarchal view of menu trees is elegant and straightforward enough, however, some of the data within the various drawers can be cryptic. As the line between Objective-C/Cocoa programming and AppleScript becomes more blurred, developers using these respective languages are sharing tools. This means scripters are coming into contact with Obj-C methods and terminology which are foreign to us. UI Browser is no exception. Bill Cheeseman says this is because UIB was written as a developer's tool for the accessibility industry. However, he also suggests a future release of UIB will probably switch to AppleScript terminology.

Let's hope that 'probably' turns more solid and sooner rather than later.

As feature requests go, I'd love to see UI Browser include Drag-and-Drop support. This might manifest in the ability to drag and drop an item within the menu hierarchy from the UIB window right into, say, a Script Editor window automatically pasting in the desired syntax. This behavior is demonstrated beautifully by Late Night Software in Script Debugger's dictionary Explorer.

GUI scripting opens incredible new worlds for OS X scripters. UI Browser is set to make those worlds easier to explore.


T.J.
tj@tjmahaffey.com

Offline

 

Board footer

Powered by FluxBB

[ Generated in 0.026 seconds, 8 queries executed ]

RSS (new topics) RSS (active topics)