simulating mouse clicks macos

Hi all,

I am trying to simulate mouse clicks via my script editor (using macOS High Sierra).
Going through the forums on the internet, nothing really worked.

Can someone help here ?

Thx in advance,
Samuel

Model: MacBook Air
Browser: Safari 537.36
Operating System: macOS 10.13

Hi and welcome to MacScripter

A few suggestions to start:

System events can click, but finding the co-ordinates can be a bit hit and miss, using code something like this:

 tell application "System Events"
	click at {100, 100}
end tell

Cliclick from https://www.bluem.net/en/projects/cliclick/ is very good, using code
something like this, the “dc” stands for double click.

 set cliclick_Path to POSIX path of (((path to desktop) as text) & "cliclick")
set cliclick_Path_Posix to POSIX path of cliclick_Path
do shell script quoted form of cliclick_Path_Posix & space & "dc:100,100" -- change the x & y coodinates

Mouse Tools from here, will help getting the coordinates of the mouse cursor, which you can then use to put into ether of the above

http://www.hamsoftengineering.com/codeSharing/MouseTools/MouseTools.html

Hi Budgie,

Thx for your repy.
I was trying to run your 1st sript;
tell application “System Events”
click at {100, 100}
end tell

But nothing happened.

Also when i was trying to run the 2nd in the following way:
tell application “Google Chrome”
activate
delay 7
end tell

set cliclick_Path to POSIX path of (((path to desktop) as text) & “cliclick”)
set cliclick_Path_Posix to POSIX path of cliclick_Path
do shell script quoted form of cliclick_Path_Posix & space & “dc:100,100” – change the x & y coodinates

I got the following error;
error “sh: /Users/Inon/Desktop/cliclick: No such file or directory” number 127

What do I miss here ?

Thx in advance,
Samuel

hi

this definitely works, you may not have seen anything happen because you don’t have anything sitting at those coordinates, try running mouse tools to get the coordiantes of your mouse cursor so it clicks on say the collapse button in a window.

tell application "System Events"
   click at {100, 100}
end tell

for the second issue, once you dowloaded the cliclick dmg and opened it, did you put the downloaded cliclick executable on your desktop?, as the script works fine.

Hi

I was trying this script (by using an excell sheet as a grid);
tell application “Microsoft Excel”
activate
delay 2
end tell

tell application “System Events”
repeat 10 times
click at {400, 100}
end repeat
end tell

tell application “System Events”
repeat 10 times
click at {800, 1300}
end repeat
end tell

tell application “System Events”
repeat 10 times
click at {1400, 1400}
end repeat
end tell

but i got this error;

error “System Events got an error: AppleEvent timed out.” number -1712

can you plz expalin what is wrong ?

thx in advance,
Samuel

Hi
Try telling system events to tell process “Microsoft Excel”

activate application "Microsoft Excel"
delay 2
tell application "System Events" to tell process "Microsoft Excel"
	
	repeat 10 times
		click at {400, 100}
	end repeat
	
	repeat 10 times
		click at {800, 1300}
	end repeat
	
	repeat 10 times
		click at {1400, 1400}
	end repeat
	
end tell 

hi

& sorry for my late reply…for some reason what you suggested is not working for me neither.

Anything else i can try ?

Thx in advance

Things that can click from Applescript you may try:

Ones it looks like you’ve tried:

  • System Events
  • ClickClick

Things you might still try:

Thx t.spoon,

I have managed to simulate clicks by python, but now i have to integrate the apple script simulation with the python.

Thats a challenge for me since I need to sync those two scripts now…
Any clue how I do that ?

Thx in advance,
Samuel

hi
try something like this

do shell script "python /Users/Samuel_Jr/Desktop/YourScriptName.py"

Samuel_Jr,

Sorry, not sure I’m understanding the problem… the example script I linked to made the Python an Applescript handler for click-and-drag operations. Doesn’t making the python into an Applescript handler integrate the two for you?

Dear t.spoon,

Thank you for your reply, i was sure my inquiry was no longer valid.
Lets make it simple… i need to insert mouse click events to my script below;

script starts;

tell application “TextEdit” --{ ################### Text Editor | NotePad a #########}
activate
delay 9
end tell

tell application “System Events”
repeat 30 times
delay (random number from 0.2 to 0.9)
key code 125 – {arrow down}
end repeat
end tell

tell application “System Events”
repeat 14 times
delay (random number from 0.2 to 0.9)
key code 126 – {arrow up}
end repeat
end tell

tell application “System Events”
repeat 8 times
delay (random number from 1.9 to 3.49)
key code 51 – {delete}
end repeat
end tell

tell application “System Events”
delay (random number from 10 to 60)
set textBuffer to " Hello "
repeat with i from 1 to count characters of textBuffer
keystroke (character i of textBuffer)
delay (random number from 1 to 5)
end repeat
end tell

tell application “System Events”
repeat 8 times
delay (random number from 1.9 to 3.49)
key code 51 – {delete}
end repeat
end tell

script ends

Your kind reply will be much appreciated.

Thank you in advance,
Samuel_jr

  1. No need all that stuff. Press Shift+Command+4 and you have live mouse position coordinates. For exit press Esc

  2. Other note is that: System Events click differs from real mouse click. It just selects clicked object on screen. For real click event ((which activates some window or its object) you must use extra tools as ClickClick or MouseTools. AppleScript without extra tools can’t do that as is modal interpretator

  3. System Events click gives real click behavior only if you click some object which is UI element , for example, some button or some menu item.

You can find which UI Elements has some Window in following way:

tell application "System Events"
    tell process "Safari"
        tell window 1
         UI Elements
     end tell
end tell
end tell

If some object you need isn’t UI Element, you must use extra tool. No System Events click at {x,y}!!!. For example, you need mouse click on certain row of window’s list and this row isn’t UI Element

Remember: for System Events, there are only windows, menu bars and buttons. Only these things it understands. And… only visible buttons…

First install MouseTools application in Applications folder

then try this:


set mouseToolsPath to (((path to applications folder) as string) & "MouseTools") as alias
set myString to quoted form of POSIX path of mouseToolsPath & " -x "

tell application "Microsoft Excel" to activate
tell application "System Events" to tell process "Microsoft Excel"
	repeat until (window "Open new and recent files" exists)
		delay 0.1
	end repeat
	do shell script myString & "250 -y 250 -doubleLeftClick"
end tell

NOTE: This script auto opens on my Mac NEW BLANK WORKBOOK - (EXCEL window at MAXIMUM size)

UI Elements of window “New and Recent Files” of process “Microsoft Excel” of application “System Events” (you can click them without extra tool, using click at {x,y} of System Events):

{menu button “Sign in” of window “Open new and recent files” of application process “Microsoft Excel” of application “System Events”, radio group 1 of window “Open new and recent files” of application process “Microsoft Excel” of application “System Events”, group 1 of window “Open new and recent files” of application process “Microsoft Excel” of application “System Events”, text field 1 of window “Open new and recent files” of application process “Microsoft Excel” of application “System Events”, group 2 of window “Open new and recent files” of application process “Microsoft Excel” of application “System Events”, button “Cancel” of window “Open new and recent files” of application process “Microsoft Excel” of application “System Events”, button “Create” of window “Open new and recent files” of application process “Microsoft Excel” of application “System Events”, button 3 of window “Open new and recent files” of application process “Microsoft Excel” of application “System Events”, button 4 of window “Open new and recent files” of application process “Microsoft Excel” of application “System Events”, button 5 of window “Open new and recent files” of application process “Microsoft Excel” of application “System Events”, toolbar 1 of window “Open new and recent files” of application process “Microsoft Excel” of application “System Events”}

Dears,

Your help is much appreciated.

as for your inputs;

when trying to write a simple script of:
Applescript:
tell application “System Events”
click at {100, 100}
end tell

it rund - nothing happens - no clicks.

Cliclick from https://www.bluem.net/en/projects/cliclick/ is like creating a sript via python - which is not useful, since i need to sync the clicks with other commands e.g writing text.

it looks so simple and but i start to wonder if i need to upgrade a version or so

  • plz bear in mind I am not a javascript pro

thx
Samuel_jr

Two things:

  1. Excel and Text Edit both have Applescript Dictionaries, so many things can be scripted directly rather than with UI scripting. What does your script actually do? Maybe you don’t need mouse clicks?

  2. I modified the Python “drag” script I’d linked to before to simply click.


on mouseClick(x, y)
	-- delayTime because the drag may fail if the UI isn't fast enough without a delay. For what I do, .1 works.
	do shell script "
	
/usr/bin/python <<END


from Quartz.CoreGraphics import CGEventCreateMouseEvent
from Quartz.CoreGraphics import CGEventCreate
from Quartz.CoreGraphics import CGEventPost
from Quartz.CoreGraphics import kCGEventLeftMouseDown
from Quartz.CoreGraphics import kCGEventLeftMouseUp
from Quartz.CoreGraphics import kCGMouseButtonLeft
from Quartz.CoreGraphics import kCGHIDEventTap
import time

def mouseEvent(type, posx, posy):

          theEvent = CGEventCreateMouseEvent(None, type, (posx,posy), kCGMouseButtonLeft)

          CGEventPost(kCGHIDEventTap, theEvent)

def mousemove(posx,posy):

          mouseEvent(kCGEventMouseMoved, posx,posy);

def mousedown(posxdown,posydown):

          mouseEvent(kCGEventLeftMouseDown, posxdown,posydown);

def mouseup(posxup,posyup):

      mouseEvent(kCGEventLeftMouseUp, posxup,posyup);

ourEvent = CGEventCreate(None);

mousedown(" & x & "," & y & ");
time.sleep(" & (".1") & ");
mouseup(" & x & "," & y & ");

END"
end mouseClick


So, you didn’t say when and where you need to click, but inserting some random clicks into your script it looks like this:


tell application "TextEdit" --{ ################### Text Editor | NotePad a #########}
	activate
	delay 9
end tell


tell application "System Events"
	repeat 30 times
		delay (random number from 0.2 to 0.9)
		key code 125 -- {arrow down}
		my mouseClick(1000, 50)
	end repeat
end tell

tell application "System Events"
	repeat 14 times
		delay (random number from 0.2 to 0.9)
		key code 126 -- {arrow up}
		my mouseClick(20, 360)
	end repeat
end tell


tell application "System Events"
	repeat 8 times
		delay (random number from 1.9 to 3.49)
		key code 51 -- {delete}
		my mouseClick(42, 42)
	end repeat
end tell


tell application "System Events"
	delay (random number from 10 to 60)
	set textBuffer to " Hello "
	repeat with i from 1 to count characters of textBuffer
		keystroke (character i of textBuffer)
		delay (random number from 1 to 5)
	end repeat
end tell



tell application "System Events"
	repeat 8 times
		delay (random number from 1.9 to 3.49)
		key code 51 -- {delete}
	end repeat
end tell


mouseClick(1342, 91)


on mouseClick(x, y)
	-- delayTime because the drag may fail if the UI isn't fast enough without a delay. For what I do, .1 works.
	do shell script "
	
/usr/bin/python <<END


from Quartz.CoreGraphics import CGEventCreateMouseEvent
from Quartz.CoreGraphics import CGEventCreate
from Quartz.CoreGraphics import CGEventPost
from Quartz.CoreGraphics import kCGEventLeftMouseDown
from Quartz.CoreGraphics import kCGEventLeftMouseUp
from Quartz.CoreGraphics import kCGMouseButtonLeft
from Quartz.CoreGraphics import kCGHIDEventTap
import time

def mouseEvent(type, posx, posy):

          theEvent = CGEventCreateMouseEvent(None, type, (posx,posy), kCGMouseButtonLeft)

          CGEventPost(kCGHIDEventTap, theEvent)

def mousemove(posx,posy):

          mouseEvent(kCGEventMouseMoved, posx,posy);

def mousedown(posxdown,posydown):

          mouseEvent(kCGEventLeftMouseDown, posxdown,posydown);

def mouseup(posxup,posyup):

      mouseEvent(kCGEventLeftMouseUp, posxup,posyup);

ourEvent = CGEventCreate(None);

mousedown(" & x & "," & y & ");
time.sleep(" & (".1") & ");
mouseup(" & x & "," & y & ");

END"
end mouseClick

Also, please use the Applescript tags when posting code. Button is just above text box when composing a comment.

If there is no active element at this point of the {100,100} screen, then nothing should happen. Only the mouse pointer should go to this point. In order for a real action to take place, there must be an element at this point on the screen that is activated when you click the mouse manually. If nothing happens when the mouse is clicked manually, then nothing will happen with a programmable click. You have already explained everything quite clearly …

Here, click this WORD. Nothing will happen (either manually or programmatically), as this is an inactive element of this web page. And now, click this WORD. This will show you how I translated to you everything I wrote here, from my Russian to English. This is active element I created on this web page for you and now you can click it manually or programmatically.

Now, as I note above, System Events’s click at {x,y} works directly only for UI elements it can see. For example, menu “History” is one UI Element of “Safari”, so we can click it directly:


tell application "Safari"
	activate
	open location "https://macscripter.net/viewtopic.php?id=46612"
end tell

tell application "System Events" to tell process "Safari"
	-- waiting to open web page completly
	repeat until (window "MacScripter / simulating mouse clicks macos" exists)
		delay 0.1
	end repeat
	-- opening menu "History"
	click at {294, 10}
end tell

Now, say I want click profile of user Samuel_Jr (is at {200,105} when this page opens in Safari in FULL SCREEN MODE). It is active element of web page, but isn’t UI Element. So here works System Events’s variant to click with waiting for focus (quit before the Safari, only to standartize this test):


tell application "Safari"
	activate
	open location "https://www.macscripter.net/viewtopic.php?pid=194441#p194441"
end tell

tell application "System Events" to tell process "Safari"
	-- wait to open page competly
	repeat until (window "MacScripter / simulating mouse clicks macos" exists)
		delay 0.1
	end repeat
	-- open page in FULL SCREEN MODE, to standartize this test
	tell window "MacScripter / simulating mouse clicks macos" to click UI element 3
	click at {200, 105}
	delay 1 -- wait to focus the element
	click at {200, 105}
end tell