Need help making GUI scripting AppleScript robust

Hi folks, I wrote an AppleScript to add attachments to a note in Notes.app at any arbitrary point in the body of the note. It works reliably in my testing so far. But it relies on GUI scripting like setting a window to a specific size and then clicking on specific coordinates. That seems brittle, and I’m worried about it breaking as its run on other machines.

Can you suggest ways to make the script more robust?

Here’s the code, and below that is an explanation and other things that I tried.

-- path to the file
set theFile to POSIX file "{full_path_to_file}"

-- name of the note to modify
set theNoteID to "{note_id}"

-- the string in the note where you want to place the cursor
set theString to "{placeholder_text}"

-- open the note and place the cursor
tell application "Notes"
	activate
	tell note id theNoteID
		show
	end tell
	set bounds of front window to {{221, 68, 1961, 1118}}
end tell

-- Find the placeholder text and leave it in a state where it's highlighted/selected and in focus.
tell application "System Events"
	tell process "Notes"
		set frontmost to true
		delay 1 -- Wait for Notes to become active
		keystroke "f" using {{command down}} -- Activate the find function
		delay 1
		keystroke theString -- find your specific text string
		keystroke return
		delay 1
		click at {{1841, 128}} -- Click the Done button on the find toolbar to dismiss it
		delay 1
	end tell
end tell

-- copy the file to the clipboard
tell application "Finder"
	set the clipboard to theFile
end tell

-- paste the file at the cursor location
tell application "System Events"
	tell process "Notes"
		keystroke "v" using {{command down}}
		delay 1
	end tell
end tell

The code is designed to be driven from Python, thus the “{}” around placeholders and the double-{} in certain spots.

Upstream of this script I have other code that assembles the HTML of the new note that I want to create. Wherever there are files or images in the HTML I convert them to placeholder text and keep track of the appropriate attachment, and then create the new note using a separate AppleScript.

Then this script runs and 1) uses command-F to find the placeholder text and highlight it, and 2) copy and paste the file or image on top of the placeholder. This results in the file or image appearing in the exact right place in the document with none of the artifacts mentioned in the post How can I attach a PDF to a note in Notes.app?, which I still personally experienced when trying that solution on Ventura 13.4.

It works reliably for me but it seems fragile. I tried addressing the “Done” button directly (the one that appears on the find toolbar), and can see info about it in Accessibility Inspector, but nothing I tried worked.

Any help is much appreciated!

1 Like

IMHO, Notes.app’s dictionary is weak when it comes to manipulating notes. There isn’t any direct support, for example, for inserting attachments at a given point, and even examining an existing note with attachments doesn’t provide a clue you can work backwards from (the body, for example, shows the HTML code for the note, but doesn’t indicate any attachments.

That pretty much leaves you with the Ui scripting approach, which is, you correctly surmise, prone to error.

On that basis, I looked for a completely different approach - namely constructing the body outside of Notes.app itself, in an app that is more AppleScript-friendly, and then setting the note’s body appropriately.
I’m poking this approach, but there is some inconsistency in how Notes handles attachments (specifically images), since sometimes they’re inline and sometimes they are attachments. Not sure if the distinction matters to you or not…?

1 Like

Totally agreed that Notes.app’s scripting dictionary is weak, and furthermore it doesn’t seem to have gained much new functionality in the past many years unfortunately.

I’m definitely interested in your approach! Can you share more details? I need to handle all file types as attachments, but an approach for images specifically that’s less brittle might be worth it.

I found it’s possible to Base64 encode the image, as explained in Base64 Image not displaying as an image, but I haven’t tried this myself and from that thread it seems to not be perfectly reliable.

Here is a better version of your script. I try very hard to never use the keystroke command or the key code command. Many times issues pop up of them being sent to wrong App.

-- path to the file
set theFile to POSIX file "{full_path_to_file}"

-- name of the note to modify
set theNoteID to "{note_id}"

-- the string in the note where you want to place the cursor
set theString to "{placeholder_text}"

-- open the note and place the cursor
tell application "Notes"
	activate
	if (index of window "Notes") > 1 then set index of window "Notes" to 1
	tell note id theNoteID
		show
	end tell
	set bounds of front window to {{221, 68, 1961, 1118}}
end tell

tell application "System Events"
	tell process "Notes"
		click menu item "Find…" of menu "Find" of menu item "Find" of menu "Edit" of menu bar 1
		repeat until exists text field 1 of scroll area 2 of splitter group 1 of window "Notes"
			delay 0.2
		end repeat
		set value of text field 1 of scroll area 2 of splitter group 1 of window "Notes" to theString -- find your specific text string
		delay 0.5
		click button "Done" of scroll area 2 of splitter group 1 of window "Notes" -- Click the Done button on the find toolbar to dismiss it
	end tell
end tell

-- copy the file to the clipboard
set the clipboard to theFile -- no need to tell Finder

-- paste the file at the cursor location
tell application "System Events"
	tell process "Notes"
		click menu item "Paste" of menu "Edit" of menu bar 1
	end tell
end tell

Here is a slightly different version that checks to see if a selection was made before pasting the file.

-- path to the file
set theFile to POSIX file "{full_path_to_file}"

-- name of the note to modify
set theNoteID to "{note_id}"

-- the string in the note where you want to place the cursor
set theString to "{placeholder_text}"

-- open the note and place the cursor
tell application "Notes"
	activate
	if (index of window "Notes") > 1 then set index of window "Notes" to 1
	tell note id theNoteID
		show
	end tell
	set bounds of front window to {{221, 68, 1961, 1118}}
end tell

-- Find the placeholder text and leave it in a state where it's highlighted/selected and in focus.
tell application "System Events"
	tell process "Notes"
		click menu item "Find…" of menu "Find" of menu item "Find" of menu "Edit" of menu bar 1
		repeat until exists text field 1 of scroll area 2 of splitter group 1 of window "Notes"
			delay 0.2
		end repeat
		set value of text field 1 of scroll area 2 of splitter group 1 of window "Notes" to theString -- find your specific text string
		delay 0.5
		click button "Done" of scroll area 2 of splitter group 1 of window "Notes" -- Click the Done button on the find toolbar to dismiss it
		repeat with i from 20 to 0 by -1 -- better timeout function
			set STRs to value of attribute "AXSelectedTextRange" of text area 1 of scroll area 2 of splitter group 1 of window "Notes"
			if ((item 2 of STRs) - (item 1 of STRs) + 1) > 0 then exit repeat
			delay 0.2
		end repeat
		if i > 0 then
			tell me
				set the clipboard to theFile -- copy the file to the clipboard
				delay 0.2
			end tell
			click menu item "Paste" of menu "Edit" of menu bar 1 -- paste the file at the cursor location
		end if
	end tell
end tell

** EDIT ** – put in 10 seconds timeout for selection

Thanks for your suggestions! I’ve been testing out the changes in your scripts, a couple notes so far.

  • For setting the clipboard to the contents of a file, I do need to wrap it in a “tell application Finder” block. If I don’t the clipboard never gets set to the file and the script pastes nothing.

  • This line doesn’t click on the Done button, nothing happens when this line runs:

click button "Done" of scroll area 2 of splitter group 1 of window "Notes" -- Click the Done button on the find toolbar to dismiss it

I’m on Venura 13.4 if that makes a difference.

Weird, it works perfect for me

I’m also on Ventura 13.4.1

Can you attach a screen capture of the Notes window with the “Done” button showing?

p.s. I edited scripts above to make sure the “Notes” window is frontmost

** EDIT ** - I edited the script above again

Thanks a ton @robertfern I’m going to give your edits a go.

Here’s a screenshot of Notes showing the “Done” button.