Thursday, August 22, 2019

#1 2018-12-05 03:22:13 am

Samuel_Jr
Member
Registered: 2018-12-04
Posts: 7

simulating mouse clicks macos

Hi all,

I am trying to simulate mouse clicks via my  script editor (using macOS High Sierra).
Going through the forums on the internet, nothing really worked.

Can someone help here ?


Thx in advance,
Samuel

Model: MacBook Air
Browser: Safari 537.36
Operating System: macOS 10.13

Offline

 

#2 2018-12-05 03:32:21 pm

Budgie
Member
From:: New Zealand
Registered: 2005-02-22
Posts: 698

Re: simulating mouse clicks macos

Hi and welcome to MacScripter

A few suggestions to start:

System events can click, but finding the co-ordinates can be a bit hit and miss, using code something like this:

Applescript:

tell application "System Events"
   click at {100, 100}
end tell

Cliclick from  https://www.bluem.net/en/projects/cliclick/ is very good, using code
something like this, the "dc" stands for double click.

Applescript:

set cliclick_Path to POSIX path of (((path to desktop) as text) & "cliclick")
set cliclick_Path_Posix to POSIX path of cliclick_Path
do shell script quoted form of cliclick_Path_Posix & space & "dc:100,100" -- change the x & y coodinates

Mouse Tools from here, will help getting the coordinates of the mouse cursor, which you can then use to put into ether of the above

http://www.hamsoftengineering.com/codeS … Tools.html

Last edited by Budgie (2018-12-05 06:30:49 pm)


Cheers
Budgie

"There's no shame in accepting aid from others... but perhaps there is a  responsibility."    - Batman

Offline

 

#3 2018-12-06 04:27:51 pm

Samuel_Jr
Member
Registered: 2018-12-04
Posts: 7

Re: simulating mouse clicks macos

Hi Budgie,

Thx for your repy.
I was trying to run your 1st sript;
tell application "System Events"
   click at {100, 100}
end tell

But nothing happened.

Also when i was trying to run the 2nd in the following way:
tell application "Google Chrome"
    activate
    delay 7
end tell

set cliclick_Path to POSIX path of (((path to desktop) as text) & "cliclick")
set cliclick_Path_Posix to POSIX path of cliclick_Path
do shell script quoted form of cliclick_Path_Posix & space & "dc:100,100" -- change the x & y coodinates


I got the following error;
error "sh: /Users/Inon/Desktop/cliclick: No such file or directory" number 127


What do I miss here ?


Thx in advance,
Samuel

Offline

 

#4 2018-12-06 04:36:36 pm

Budgie
Member
From:: New Zealand
Registered: 2005-02-22
Posts: 698

Re: simulating mouse clicks macos

hi

this definitely works, you may not have seen anything happen because you don't have anything sitting at those coordinates, try running mouse tools to get the coordiantes of your mouse cursor so it clicks on say the collapse button in a window.

Applescript:

tell application "System Events"
click at {100, 100}
end tell

for the second issue, once you dowloaded the cliclick dmg and opened it, did you put the downloaded cliclick executable on your desktop?, as the script works fine.


Cheers
Budgie

"There's no shame in accepting aid from others... but perhaps there is a  responsibility."    - Batman

Offline

 

#5 2018-12-07 01:39:14 pm

Samuel_Jr
Member
Registered: 2018-12-04
Posts: 7

Re: simulating mouse clicks macos

Hi

I was trying this script (by using an excell sheet as a grid);
tell application "Microsoft Excel"
    activate
    delay 2
end tell


tell application "System Events"
    repeat 10 times
        click at {400, 100}
    end repeat
end tell

tell application "System Events"
    repeat 10 times
        click at {800, 1300}
    end repeat
end tell

tell application "System Events"
    repeat 10 times
        click at {1400, 1400}
    end repeat
end tell



but i got this error;

error "System Events got an error: AppleEvent timed out." number -1712



can you plz expalin what is wrong ?


thx in advance,
Samuel

Offline

 

#6 2018-12-10 03:18:03 pm

Budgie
Member
From:: New Zealand
Registered: 2005-02-22
Posts: 698

Re: simulating mouse clicks macos

Hi
Try telling system events to tell process "Microsoft Excel"

Applescript:

activate application "Microsoft Excel"
delay 2
tell application "System Events" to tell process "Microsoft Excel"
   
   repeat 10 times
       click at {400, 100}
   end repeat
   
   repeat 10 times
       click at {800, 1300}
   end repeat
   
   repeat 10 times
       click at {1400, 1400}
   end repeat
   
end tell

Last edited by Budgie (2018-12-11 07:18:09 pm)


Cheers
Budgie

"There's no shame in accepting aid from others... but perhaps there is a  responsibility."    - Batman

Offline

 

#7 2018-12-17 03:09:30 pm

Samuel_Jr
Member
Registered: 2018-12-04
Posts: 7

Re: simulating mouse clicks macos

hi

& sorry for my late reply.....for some reason what you suggested is not working for me neither.

Anything else i can try ?



Thx in advance

Offline

 

#8 2018-12-19 09:12:48 am

t.spoon
Member
From:: BFE, Massachusetts
Registered: 2013-01-13
Posts: 411

Re: simulating mouse clicks macos

Things that can click from Applescript you may try:

Ones it looks like you've tried:
- System Events
- ClickClick

Things you might still try:
- AS Toolbox  https://astoolbox.wordpress.com (note that, I've heard that as of Mojave, OSAXen must be script embedded, not installed system-wide)
- MouseTools http://www.hamsoftengineering.com/codeS … Tools.html
- Python https://macscripter.net/viewtopic.php?id=46154


Hackintosh built February, 2012 |  Mac OS Sierra
GIGABYTE GA-Z68X-UD3H-B3 | Core i5 2500k | 16 GB DDR3 | GIGABYTE Geforce 1050 TI 4GB
250 GB Samsung 850 EVO | 4 TB RAID
Dell Ultrasharp U3011 | Dell Ultrasharp 2007FPb

Offline

 

#9 2018-12-22 04:14:01 am

Samuel_Jr
Member
Registered: 2018-12-04
Posts: 7

Re: simulating mouse clicks macos

Thx t.spoon,

I have managed to simulate clicks by python, but now i have to integrate the apple script simulation with the python.


Thats a challenge for me since I need to sync those two scripts now......
Any clue how I do that ?



Thx in advance,
Samuel

Offline

 

#10 2018-12-26 01:50:22 pm

Budgie
Member
From:: New Zealand
Registered: 2005-02-22
Posts: 698

Re: simulating mouse clicks macos

hi
try something like this

Applescript:

do shell script "python /Users/Samuel_Jr/Desktop/YourScriptName.py"


Cheers
Budgie

"There's no shame in accepting aid from others... but perhaps there is a  responsibility."    - Batman

Offline

 

#11 2018-12-26 02:25:57 pm

t.spoon
Member
From:: BFE, Massachusetts
Registered: 2013-01-13
Posts: 411

Re: simulating mouse clicks macos

Samuel_Jr,

but now i have to integrate the apple script simulation with the python



Sorry, not sure I'm understanding the problem... the example script I linked to made the Python an Applescript handler for click-and-drag operations. Doesn't making the python into an Applescript handler integrate the two for you?


Hackintosh built February, 2012 |  Mac OS Sierra
GIGABYTE GA-Z68X-UD3H-B3 | Core i5 2500k | 16 GB DDR3 | GIGABYTE Geforce 1050 TI 4GB
250 GB Samsung 850 EVO | 4 TB RAID
Dell Ultrasharp U3011 | Dell Ultrasharp 2007FPb

Offline

 

#12 2019-05-01 12:31:03 pm

Samuel_Jr
Member
Registered: 2018-12-04
Posts: 7

Re: simulating mouse clicks macos

Dear t.spoon,

Thank you for your reply, i was sure my inquiry was no longer valid.
Lets make it simple... i need to insert mouse click events to my script below;

script starts;

tell application "TextEdit" --{ ################### Text Editor | NotePad a #########}
    activate
    delay 9
end tell


tell application "System Events"
    repeat 30 times
        delay (random number from 0.2 to 0.9)
        key code 125 -- {arrow down}
    end repeat
end tell

tell application "System Events"
    repeat 14 times
        delay (random number from 0.2 to 0.9)
        key code 126 -- {arrow up}
    end repeat
end tell


tell application "System Events"
    repeat 8 times
        delay (random number from 1.9 to 3.49)
        key code 51 -- {delete}
    end repeat
end tell


tell application "System Events"
    delay (random number from 10 to 60)
    set textBuffer to " Hello "
    repeat with i from 1 to count characters of textBuffer
        keystroke (character i of textBuffer)
        delay (random number from 1 to 5)
    end repeat
end tell



tell application "System Events"
    repeat 8 times
        delay (random number from 1.9 to 3.49)
        key code 51 -- {delete}
    end repeat
end tell




script ends



Your kind reply will be much appreciated.

Thank you in advance,
Samuel_jr

Offline

 

#13 2019-05-01 09:47:27 pm

KniazidisR
Member
Registered: 2019-03-03
Posts: 427

Re: simulating mouse clicks macos

Budgie wrote:

Hi and welcome to MacScripter

A few suggestions to start:

System events can click, but finding the co-ordinates can be a bit hit and miss, using code something like this:

Applescript:

tell application "System Events"
   click at {100, 100}
end tell

Cliclick from  https://www.bluem.net/en/projects/cliclick/ is very good, using code
something like this, the "dc" stands for double click.

Applescript:

set cliclick_Path to POSIX path of (((path to desktop) as text) & "cliclick")
set cliclick_Path_Posix to POSIX path of cliclick_Path
do shell script quoted form of cliclick_Path_Posix & space & "dc:100,100" -- change the x & y coodinates

Mouse Tools from here, will help getting the coordinates of the mouse cursor, which you can then use to put into ether of the above

http://www.hamsoftengineering.com/codeS … Tools.html



1) No need all that stuff. Press Shift+Command+4 and you have live mouse position coordinates. For exit press Esc

2) Other note is that: System Events click differs from real mouse click. It just selects clicked object on screen. For real click event ((which activates some window or its object) you must use extra tools as ClickClick or MouseTools. AppleScript without extra tools can't do that as is modal interpretator

3) System Events click gives real click behavior only if you click some object which is UI element , for example, some button or some menu item.

You can find which UI Elements has some Window in following way:

Applescript:

tell application "System Events"
tell process "Safari"
tell window 1
UI Elements
end tell
end tell
end tell

If some object you need isn't UI Element, you must use extra tool. No System Events click at {x,y}!!!. For example, you need mouse click on certain row of window's list and this row isn't UI Element

Remember: for System Events, there are only windows, menu bars and buttons. Only these things it understands. And... only visible buttons...

Last edited by KniazidisR (2019-05-01 10:46:05 pm)


macOS Mojave -- version 10.14.4
Safari -- version 12.1

Offline

 

#14 2019-05-01 11:15:20 pm

KniazidisR
Member
Registered: 2019-03-03
Posts: 427

Re: simulating mouse clicks macos

Samuel_Jr wrote:

hi

& sorry for my late reply.....for some reason what you suggested is not working for me neither.



First install MouseTools application in Applications folder

then try this:

Applescript:


set mouseToolsPath to (((path to applications folder) as string) & "MouseTools") as alias
set myString to quoted form of POSIX path of mouseToolsPath & " -x "

tell application "Microsoft Excel" to activate
tell application "System Events" to tell process "Microsoft Excel"
   repeat until (window "Open new and recent files" exists)
       delay 0.1
   end repeat
   do shell script myString & "250 -y 250 -doubleLeftClick"
end tell

NOTE: This script auto opens on my Mac  NEW BLANK WORKBOOK - (EXCEL window at MAXIMUM size)

Last edited by KniazidisR (2019-05-02 12:42:23 am)


macOS Mojave -- version 10.14.4
Safari -- version 12.1

Offline

 

#15 2019-05-02 12:59:27 am

KniazidisR
Member
Registered: 2019-03-03
Posts: 427

Re: simulating mouse clicks macos

UI Elements of window "New and Recent Files" of process "Microsoft Excel" of application "System Events" (you can click them without extra tool, using click at {x,y} of System Events):

{menu button "Sign in" of window "Open new and recent files" of application process "Microsoft Excel" of application "System Events", radio group 1 of window "Open new and recent files" of application process "Microsoft Excel" of application "System Events", group 1 of window "Open new and recent files" of application process "Microsoft Excel" of application "System Events", text field 1 of window "Open new and recent files" of application process "Microsoft Excel" of application "System Events", group 2 of window "Open new and recent files" of application process "Microsoft Excel" of application "System Events", button "Cancel" of window "Open new and recent files" of application process "Microsoft Excel" of application "System Events", button "Create" of window "Open new and recent files" of application process "Microsoft Excel" of application "System Events", button 3 of window "Open new and recent files" of application process "Microsoft Excel" of application "System Events", button 4 of window "Open new and recent files" of application process "Microsoft Excel" of application "System Events", button 5 of window "Open new and recent files" of application process "Microsoft Excel" of application "System Events", toolbar 1 of window "Open new and recent files" of application process "Microsoft Excel" of application "System Events"}


macOS Mojave -- version 10.14.4
Safari -- version 12.1

Offline

 

#16 2019-05-02 02:28:38 am

Samuel_Jr
Member
Registered: 2018-12-04
Posts: 7

Re: simulating mouse clicks macos

Dears,

Your help is much appreciated.

as for your inputs;

when trying to write a simple script of:
Applescript:
tell application "System Events"
   click at {100, 100}
end tell

it rund - nothing happens - no clicks.



Cliclick from  https://www.bluem.net/en/projects/cliclick/ is like creating a sript via python - which is not useful, since i need to sync the clicks with other commands e.g writing text.




it looks so simple and but i start to wonder if i need to upgrade a version or so

* plz bear in mind I am not a javascript pro


thx
Samuel_jr

Offline

 

#17 2019-05-02 02:57:41 pm

t.spoon
Member
From:: BFE, Massachusetts
Registered: 2013-01-13
Posts: 411

Re: simulating mouse clicks macos

Two things:

1. Excel and Text Edit both have Applescript Dictionaries, so many things can be scripted directly rather than with UI scripting. What does your script actually do? Maybe you don't need mouse clicks?

2. I modified the Python "drag" script I'd linked to before to simply click.

Applescript:


on mouseClick(x, y)
   -- delayTime because the drag may fail if the UI isn't fast enough without a delay. For what I do, .1 works.
   do shell script "
   
/usr/bin/python <<END


from Quartz.CoreGraphics import CGEventCreateMouseEvent
from Quartz.CoreGraphics import CGEventCreate
from Quartz.CoreGraphics import CGEventPost
from Quartz.CoreGraphics import kCGEventLeftMouseDown
from Quartz.CoreGraphics import kCGEventLeftMouseUp
from Quartz.CoreGraphics import kCGMouseButtonLeft
from Quartz.CoreGraphics import kCGHIDEventTap
import time

def mouseEvent(type, posx, posy):

theEvent = CGEventCreateMouseEvent(None, type, (posx,posy), kCGMouseButtonLeft)

CGEventPost(kCGHIDEventTap, theEvent)

def mousemove(posx,posy):

mouseEvent(kCGEventMouseMoved, posx,posy);

def mousedown(posxdown,posydown):

mouseEvent(kCGEventLeftMouseDown, posxdown,posydown);

def mouseup(posxup,posyup):

mouseEvent(kCGEventLeftMouseUp, posxup,posyup);

ourEvent = CGEventCreate(None);

mousedown("
& x & "," & y & ");
time.sleep("
& (".1") & ");
mouseup("
& x & "," & y & ");

END"

end mouseClick

So, you didn't say when and where you need to click, but inserting some random clicks into your script it looks like this:

Applescript:


tell application "TextEdit" --{ ################### Text Editor | NotePad a #########}
   activate
   delay 9
end tell


tell application "System Events"
   repeat 30 times
       delay (random number from 0.2 to 0.9)
       key code 125 -- {arrow down}
       my mouseClick(1000, 50)
   end repeat
end tell

tell application "System Events"
   repeat 14 times
       delay (random number from 0.2 to 0.9)
       key code 126 -- {arrow up}
       my mouseClick(20, 360)
   end repeat
end tell


tell application "System Events"
   repeat 8 times
       delay (random number from 1.9 to 3.49)
       key code 51 -- {delete}
       my mouseClick(42, 42)
   end repeat
end tell


tell application "System Events"
   delay (random number from 10 to 60)
   set textBuffer to " Hello "
   repeat with i from 1 to count characters of textBuffer
       keystroke (character i of textBuffer)
       delay (random number from 1 to 5)
   end repeat
end tell



tell application "System Events"
   repeat 8 times
       delay (random number from 1.9 to 3.49)
       key code 51 -- {delete}
   end repeat
end tell


mouseClick(1342, 91)


on mouseClick(x, y)
   -- delayTime because the drag may fail if the UI isn't fast enough without a delay. For what I do, .1 works.
   do shell script "
   
/usr/bin/python <<END


from Quartz.CoreGraphics import CGEventCreateMouseEvent
from Quartz.CoreGraphics import CGEventCreate
from Quartz.CoreGraphics import CGEventPost
from Quartz.CoreGraphics import kCGEventLeftMouseDown
from Quartz.CoreGraphics import kCGEventLeftMouseUp
from Quartz.CoreGraphics import kCGMouseButtonLeft
from Quartz.CoreGraphics import kCGHIDEventTap
import time

def mouseEvent(type, posx, posy):

theEvent = CGEventCreateMouseEvent(None, type, (posx,posy), kCGMouseButtonLeft)

CGEventPost(kCGHIDEventTap, theEvent)

def mousemove(posx,posy):

mouseEvent(kCGEventMouseMoved, posx,posy);

def mousedown(posxdown,posydown):

mouseEvent(kCGEventLeftMouseDown, posxdown,posydown);

def mouseup(posxup,posyup):

mouseEvent(kCGEventLeftMouseUp, posxup,posyup);

ourEvent = CGEventCreate(None);

mousedown("
& x & "," & y & ");
time.sleep("
& (".1") & ");
mouseup("
& x & "," & y & ");

END"

end mouseClick

Also, please use the Applescript tags when posting code. Button is just above text box when composing a comment.

Last edited by t.spoon (2019-05-02 02:58:42 pm)


Hackintosh built February, 2012 |  Mac OS Sierra
GIGABYTE GA-Z68X-UD3H-B3 | Core i5 2500k | 16 GB DDR3 | GIGABYTE Geforce 1050 TI 4GB
250 GB Samsung 850 EVO | 4 TB RAID
Dell Ultrasharp U3011 | Dell Ultrasharp 2007FPb

Offline

 

#18 2019-05-04 05:07:56 pm

KniazidisR
Member
Registered: 2019-03-03
Posts: 427

Re: simulating mouse clicks macos

Samuel_Jr wrote:

Hi Budgie,

Thx for your repy.
I was trying to run your 1st sript;
tell application "System Events"
   click at {100, 100}
end tell

But nothing happened.

What do I miss here ?


Thx in advance,
Samuel



If there is no active element at this point of the {100,100} screen, then nothing should happen. Only the mouse pointer should go to this point. In order for a real action to take place, there must be an element at this point on the screen that is activated when you click the mouse manually. If nothing happens when the mouse is clicked manually, then nothing will happen with a programmable click. You have already explained everything quite clearly ...

Here, click this WORD. Nothing will happen (either manually or programmatically), as this is an inactive element of this web page. And now, click this WORD. This will show you how I translated to you everything I wrote here, from my Russian to English. This is active element I created on this web page for you and now you can click it manually or programmatically.

Now, as I note above, System Events's click at {x,y} works directly only for UI elements it can see. For example, menu "History" is one UI Element of "Safari", so we can click it directly:

Applescript:


tell application "Safari"
   activate
   open location "https://macscripter.net/viewtopic.php?id=46612"
end tell

tell application "System Events" to tell process "Safari"
   -- waiting to open web page completly
   repeat until (window "MacScripter / simulating mouse clicks macos" exists)
       delay 0.1
   end repeat
   -- opening menu "History"
   click at {294, 10}
end tell

Now, say I want click profile of user Samuel_Jr (is at {200,105} when this page opens in Safari in FULL SCREEN MODE). It is active element of web page, but isn't UI Element. So here works System Events's variant to click with waiting for focus (quit before the Safari, only to standartize this test):

Applescript:


tell application "Safari"
   activate
   open location "https://www.macscripter.net/viewtopic.php?pid=194441#p194441"
end tell

tell application "System Events" to tell process "Safari"
   -- wait to open page competly
   repeat until (window "MacScripter / simulating mouse clicks macos" exists)
       delay 0.1
   end repeat
   -- open page in FULL SCREEN MODE, to standartize this test
   tell window "MacScripter / simulating mouse clicks macos" to click UI element 3
   click at {200, 105}
   delay 1 -- wait to focus the element
   click at {200, 105}
end tell

Last edited by KniazidisR (2019-05-04 08:00:52 pm)


macOS Mojave -- version 10.14.4
Safari -- version 12.1

Offline

 

Board footer

Powered by FluxBB

RSS (new topics) RSS (active topics)