a Halloween tale [was: Does 'run script' inhibit logging in Yosemite?]

Basically, OSA has always included APIs for implementing interactive consoles; e.g. Script Debugger and Smile have long provided this feature.

It’s pretty simple too: when the user starts a new console, call OSAMakeContext to create a persistent context object to store any session state, then each time the user types a line of code and presses enter, pass that source code and the context object to OSADoScript to execute it. Any variables/handlers defined by that line are added to the context, so can be used in subsequent lines. When the user closes the console, call OSADispose to dispose of the context object and free up the memory.

10.10osascript also adds an interactive mode, and at first glance it appears to work as advertised:

$ osascript -i -l JavaScript
>> x = 3
=> 3
>> x
=> 3

Try it with AppleScript, however, and it falls apart:

$ osascript -i
>> set x to 3
=> 3
>> x
!! The variable x is not defined.

Turns out that instead of using a persistent context, osascript compiles each line as a new script object with its own built-in context. This is why the AppleScript example fails: the built-in context is only available to that particular script object. But wait, there’s more: the reason why osascript does this is because JXA, instead of supporting the established APIs for creating and sharing persistent contexts, ‘ingeniously’ invents its own scheme whereby reusing an existing script object’s ID doesn’t actually replace the current script object (as it’s supposed to) but instead replaces the object’s existing code without clearing its context then compiles and runs that code on top of that ‘dirty’ context.

So JXA and osascript appear to support interactive operation, but they’re doing so in a way that is fundamentally broken and wrong - prima facie evidence that Apple’s current AppleScript/Automation team fundamentally does not understand how OSA works.

(Incidentally, there’s plenty other defects demonstrating they don’t understand Apple events, the Apple Event Object Mode, or even the JavaScript language either, but I don’t want to give you any more nightmares: I already sleep badly enough myself knowing my own automation scripts and systems live or die upon the whims of demonic muppets.)

I’m wondering, have you submitted your complaints as official bug reports? In my experience, the team at Apple may take ages to reply, but they do seem to take note and try to fix problems. Also, I have a feeling that the team working on scripting technologies is struggling to deliver within schedule (for instance, the AppleScript Language Guide has not been updated for OS X 10.10 yet). That’s pure speculation, of course, but the specific example you show might well be the result of a hastened implementation. After all, there have been many changes in OS X scripting technologies in Mavericks and Yosemite, and the team working on them might be relatively small.

Sorry, no, I’ve not submitted these as bug reports, and at this point I’ve no intention of doing so. I contacted Sal directly shortly after WWDC and set a whole lot of feedback on JXA, most of which they completely ignored. I even spent six weeks assembling a nearly-complete reference implementation for them to learn from or even steal outright, and didn’t even get a reply. I bent over backwards offering help when there was still a chance to get it right, and I got was smoke up my ass. So my only aim now is to what I can to make users aware they’ve been sold yet another pup.

Mind that it’s been seven years since the same team screwed up Scripting Bridge; I think it’s safe to say that not only have they failed to learn from their own mistakes, but that they’re incapable of doing so. If anything, they’ve expended more effort inventing ingenious new ways to make it work wrong than simply copying how AppleScript already does it (the correct solution). They have a product manager plus three highly-trained, highly-paid software engineers; resources I would kill for.

So if a dumb old AppleScript user like me can build a working solution in a handful of weeks with no Apple documentation, Apple source code, or any other support or assistance, what’s their excuse?

(BTW, I’m not narked at you for asking; it’s just sheer frustration at Apple in general and their Automation team in particular.)

Anyway, here’s a fresh example that demonstrates how the AS team don’t understand how JavaScript works…

What do you suppose the following JXA statements will return?


    Application('TextEdit').documents['untitled'].get()

    Application('TextEdit').documents['text'].get()

According to the JXA documentation, the above statements should be equivalent to these:


    tell app "TextEdit" to get document "untitled"

    tell app "TextEdit" to get document "text"

Now create some test documents in TextEdit, then run each JXA statement in turn. The result may come as a surprise, but I guarantee you that JavaScript is working exactly as designed, because, as anyone familiar with the language can tell you:


    someRef['aName']

is simply a synonym for:


    someRef.aName

Thus, the following statements are identical:


    Application('TextEdit').documents['text'].get()

    Application('TextEdit').documents.text.get()

As a result, the first example returns a reference to the document named ‘untitled’ as expected (since the TextEdit dictionary doesn’t define that word as a keyword):


    Application('TextEdit').documents['untitled'].get()
   --> Application('TextEdit').documents['untitled']

but the second returns a list of strings representing each document’s text:


    Application('TextEdit').documents['text'].get()
   --> ["blah blah", "blah blah blah", ...]

And if you still don’t see why this is such a problem, consider the implications for robust, predictable program behavior when the document’s name is supplied by a variable whose value could be anything:


    Application('TextEdit').documents[docName].get()

This was the very first defect I pointed out to Sal: that using […] syntax to construct by-name specifiers is completely unsafe, because it can’t reliably distinguish an object’s name from the name of one of its properties or elements. The correct solution (which I also provided) is to 1. provide a dedicated byName method for constructing by-name specifiers (which they did), and 2. avoid using […] syntax to construct by-name specifiers. (FWIW, it’s still possible to use […] syntax to construct by-index specifiers, since those are always numeric and property/element names are always non-numeric.)

It’s just a simple dumb design flaw that could’ve been fully plugged in 5 minutes months before JXA shipped, but they chose not to.

And a third example of a much less obvious but ultimately crippling defect that is essentially unfixable within the current JXA design:

I do professional Illustrator automation myself, sometimes using AppleScript for quick scripts but mostly using Python+appscript as it’s far more capable for heavy-duty work. Needless to say, when everyone’s livelihoods are relying on my automation working correctly, I have very little tolerance for failures in either my own code or anyone else’s. So you can imagine my reaction to JXA when converting the following simple save command, which works perfectly in AS:


    set theFile to "/Users/has/AS-save-test.pdf"

    tell application "Adobe Illustrator"
       save document 1 in theFile as pdf ¬
           with options {class:PDF save options, trim marks:true}
    end tell

and Python appscript:


   theFile = "/Users/has/PY-save-test.pdf"

   app('Adobe Illustrator').documents[1].save(in=theFile, as=k.pdf,
         with_options={k.class: k.PDF_save_options, k.trim_marks: True})

to JXA:


    theFile = "/Users/has/JS-save-test.pdf";

    AI = Application('Adobe Illustrator');

    AI.documents[0].save({in:theFile, as:"pdf",
           withOptions:{class:"PDFSaveOptions", trimMarks:true}});

only to get Error: Expected fileSpecification but found string.

This problem occurs because the type information provided by dictionaries is never used by AppleScript: it’s purely there for documentation purposes, so is not guaranteed to be be sufficient or correct, or even meaningful to anything but a human reader. Unfortunately, JXA uses this information to do its own fancy casting of JS values to AE types before sending the command, so whenever that dictionary-supplied information is incomplete or incorrect, or simply not one that JXA knows about, it falls over.

In this particular case, AI describes the in parameter’s type as file specification (aka typeFSS), which if you remember pre-OS X was the value type used to specify files that may or may not currently exist, but was deprecated around 10.5 as the Carbon FSSpec datatype didn’t support Unicode. Obviously, Adobe’s hacked in some updates since then, so nowadays it accepts POSIX or HFS path strings, although annoyingly it still doesn’t take POSIX file (aka typeFileURL) values, which is what it really ought to do.

Still, JXA does provide a strictParameterType option for disabling this fancy casting behavior in situations where it causes such problems:


theFile = "/Users/has/save-js.pdf";

AI = Application('Adobe Illustrator');
AI.strictParameterType = false;

AI.documents[0].save({in:theFile, as:"pdf",
        withOptions:{class:"PDFSaveOptions", trimMarks:true}});

Now the file saves… yay!!!

But wait… A new problem appears, and this is the real deal breaker: the saved file doesn’t contain trim marks or any of the other settings specified by the withOptions parameter. This is because Illustrator silently ignores the entire ‘save options’ record if isn’t in the correct format. (It really ought to report a bad parameter error, but as experienced AppleScripters you know what these old Carbon warhorses are like, and we forgive them their little quirks because without them AppleScript would be worthless.)

In this case, the record’s class property needs to contain a type name, but contains a string instead. Of course, JXA’s fancy auto-casting system was intended to convert strings to the appropriate typeType/typeEnumerator type, and in trivial use-cases - e.g. doc.close({saving:“no”}) - this mechanism works as intended. However, here the relevant string lies deep within a record, with no obvious way for JXA to determine what its type should be. Even if the strictParameterType option could be left on, the record’s values still wouldn’t be cast correctly since the AI dictionary only describes thewithOptions parameter’s type as any. And it’s tricky for the dictionary to be more specific than that, since each saved/exported document format - ai, eps, pdf, etc. - uses a different type of record.

Even if JXA could be “educated” on how to pack such complex parameters correctly, there’s zero guarantee that another application won’t do something very similar but in a slightly different way that causes JXA to puke all over again. So not only does JXA not work properly with Illustrator, the way it’s designed to work makes it incapable of ever working right, because it’ll always choke on some app somewhere.

The correct solution? Do what AppleScript does: allow the user to choose what types of objects they wish to supply, and leave the application to use/coerce/refuse those values as appropriate. They have a far better understand of what’s appropriate than the AE bridge ever will.

Of course, not every language has built-in classes suitable for representing all of the available AE types, so it may be necessary for the bridge to implement one or more custom classes. For example, Python and JavaScript don’t currently include a Symbol class suitable for representing AE type and enumerator names, so it’s necessary for the bridge to provide a suitable substitute. The JavaScriptOSA component I sent to the AS team does this: it implements a Keyword ‘class’ for representing type/enumerator/property/etc. names, plus a convenient mechanism (the k object) for creating new Keyword objects within your code. Thus, the ‘with options’ record in the original AppleScript command:


{class: PDF save options, trim marks: true}

translates unambiguously to this in JavaScriptOSA:


{class: k.PDFSaveOptions, trimMarks: true}

When that object is passed to the AE bridge, it doesn’t require any dictionary-driven guesswork to figure out what type each value should be as those values already contain all the type information required. Simple, reliable, proven.

Alas, the road to hell is paved with ingeniously clever “solutions” to hellishly convoluted problems that would never would’ve existed in the first place, had corporate hellspawn programmers not invented those problems themselves just to have something exciting to do.

Not that it detracts form your point, but it is used in Cocoa scripting.

Hi all,

What I was wondering about is what is so good about javascript anyway. Is there something special that it can do besides what you can do with what we already have? I know that you can use it in browsers on websites and such. Why did Apple add that to the scripting components again?

Just wondering,
kel

Think I know why Apple did that. Now that the Finder windows are browsers, you can use javascript to manipulate the windows. That’s why! :slight_smile: Or am I way off base on that?

Way off. The prime reason is presumably that millions of people already know javascript.

Indeed, though as you say not relevant here as CS is server-side technology while AppleScript and JXA are both clients. JXA’s type system wouldn’t be quite as problematic if CS was used by all scriptable apps (although it’d still fall down in places, e.g. when the type is defined as ‘any’, or ‘text or ’). But there’s a ton of Carbon-based apps out there whose AE support is implemented in other ways - many of which are key productivity apps - and these don’t use AETE/SDEF data at all.

That said, I do think you’re onto something. When Matt N reported the first compatibility bug between Scripting Bridge and BBEdit, Chris N’s response was basically “We designed SB to work with Cocoa apps”, which is just another way of saying we didn’t design it to work with Carbon apps. (Incidentally, SB breaks on Cocoa apps as well.) So it smells more like an ideological disagreement between Chris N and the rest of the AppleScript ecosystem - between how he believes application scripting ought to work and how it actually does [1].

Needless to say, the many hundreds of scriptable apps already out in the real world don’t give a flying crap what Chris N, SB, or JXA think; AppleScript is the de facto specification against which they have all been designed, implemented, and tested against, so if they work as designed with AppleScript then they are correct, and too bad for SB/JXA’s hapless users if they puke instead. It’s the client software’s job to ensure it’s compatible with the server, not the other way around; the fact that Apple’s own engineers don’t seem to understand this is worrisome to say the least.

[1] FWIW, my very first attempt at appscript suffered some of the same overbearing smartassery, but being an AppleScript user I realized it sucked, threw it out, and redesigned and rewrote it over again. So I understand where Chris N is coming from, and I know he’s dead wrong. Unfortunately, a lot of corporate programming culture is dysfunctional this way, not least because such programmers never use their own products (“eat their own dogfood”) themselves.

Indeedy. Originally JavaScript was locked into a single application (the web browser), but ever since NodeJS became a thing, it’s been making serious inroads into server and desktop as well, buoyed by the huge existing user base and by the massive investment by Google, Apple, etc in developing high-performance JavaScript interpreters.

It’s just a shame that JavaScript itself is only a three-fifths decent language, the other two-fifths being now-unfixable ECMA Standards-approved crap. (And oyy, that syntax…) OTOH, all mainstream languages are crappy to some degree, and AppleScript’s far from saint either, so in theory JXA should be a great addition to the OS X desktop nevertheless. Unfortunately, as I’m slowly documenting, the JXA component’s design and implementation is largely rotten, so it’s more another case of one step forward and two steps back. :frowning:

So anyway, let’s take a look at JXA’s support for reference forms, as this is one of the areas where JXA fails hardest. This’ll take a few posts to get through, so please be patient as I’ll spread them over the week. (Spoiler: it gets worse as it goes on.)

Apple event object specifiers allow the following reference forms:

Properties:

  • property
  • user property

Elements by:

  • index
  • name
  • ID
  • absolute position (first/middle/last/any/all)
  • relative position (previous/next)
  • range
  • test (aka ‘whose clause’)

Insertion location (beginning, end, before, after)

All of these forms are (natch) 100% supported by AppleScript. Let’s see how JXA compares, starting with the ‘elements by-index’ form (I’ll cover properties at the end).

By-index references are obviously supported, and use JavaScript’s zero-indexing convention (i.e. the first item is 0, the second is 1, etc.):


Application('Finder').home.folders[0].get()
--> Application("Finder").startupDisk.folders.byName("Users").folders.byName("foo").folders.byName("Documents")

There’s also an at method [1], allowing you to write:


Application('Finder').home.folders.at(0).get()

It would be much more helpful, however, if the at method adhered to Apple events’ native one-indexing. As you know, some applications such as Finder allow you to get an item’s index property:


Item‚Object : An item
	elements	
		contained by application, containers, disks, folders, desktop-objects, trash-objects.
	properties
		name (text) : the name of the item
		displayedName (text, r/o) : the user-visible name of the item
		nameExtension (text) : the name extension of the item (such as "txt")
		extensionHidden (boolean) : Is the item's extension hidden from the user?
		index (integer, r/o) : the index in the front-to-back ordering within its container
		...

AppleScript also uses one-indexing, so there’s obviously no problem round-tripping index numbers there:


tell app "Finder" to get index of item 1 of home
--> 1

Now let’s ask JXA to get the index of the first item in home:


Application('Finder').home.items.at(0).index.get()
--> 1

Slight oopsie: Apple event elements are one-indexed, so naturally the index property’s value is 1. If we want to use this index in another JXA reference, we’ll need to subtract 1 to adjust for the difference between one-indexed and zero-indexed. Unless the index is a negative number, in which case don’t subtract anything, of course.

It’s a minor irritation, but it could’ve been avoided had the at method respected Apple events conventions for compatibility, while the […] syntax shortcut provided JavaScript-style zero-indexing for convenience. And it sets a bad precedence - this mismatch between AE and JS indexing conventions styles starts to cause real pain when implementing support for by-range references.

[1] In case you’re wondering, the at method is necessary in cases where the ‘index’ value is a non-number (which is allowed by Apple events). For example, Finder also allows you to pass an alias/POSIX file object:


set aFile to alias "Macintosh HD:Users:foo"
tell app "Finder" to get item aFile
--> folder "foo" of folder "Users" of startup disk of application "Finder"

JS’s […] syntax only accepts strings though, so you can only pass the equivalent Path object via the at method:


Application('Finder').items.at(Path('/Users/foo')).get()
--> Application("Finder").startupDisk.folders.byName("Users").folders.byName("foo")

I’ve a feeling this was one of the compatibility problems I raised and which they addressed, though can’t really remember. Unfortunately, while the JXA devs tended to fix the trivial compatibility bugs, they completely ignored the non-trivial ones. And now that JXA is released, non-trivial defects will be far more difficult and painful (if not impossible) to fix in future, as making non-trivial changes often requires breaking compatibility with users’ existing scripts.

I’m not sure I find all the points you make compelling (for example, about JXA using dictionaries to infer type information being a “bad ideaâ„¢”): I’m still pondering (JXA is still new to me). But please go on, I like your deep critical assessment of some design choices, which gives a useful insight into the language.