Cycling through URL's to find download package?

@dbrewood, hello,

Your newest installer takes more than 5 seconds, so you can put timeout handling to grab it and automatically download it.

(You can check that the installer downloads indeed, with periodically opening and closing the downloads folder). Or, simply wait for script finishing.


repeat with aNumber from 2200 to 2109 by -1
	
	set MyDownloadPlace to quoted form of (POSIX path of (path to downloads folder) & "emclient-v9.1." & aNumber & "_Mac.pkg")
	set anURL to "https://cdn-dist.emclient.com/dist/v9.1." & aNumber & "_Mac/setup.pkg"
	
	try
		set maybe to do shell script "curl " & quoted form of anURL & " -m 5.0" -- timeout 5 seconds
	on error number 28
		do shell script "curl -L " & quoted form of anURL & " -o " & MyDownloadPlace
		exit repeat
	end try
	
end repeat

That is another cool solution to the problem, many thanks. I’ll test all out tomorrow as we’re about to hit family TV time :wink: [i.e. I’ve been told to get off the Mac]…

I refined it little.

I just ran the original 3 times and each time it downloaded different files, the latest one and then the other two:

emclient-v9.1.2129_Mac.pkg
emclient-v9.1.2130_Mac.pkg
emclient-v9.1.2148_Mac.pkg

I’ll try the revised version in the morning.

Weird. I expected any other error, but not this one. According to the logic of my script, this cannot be, because there is an exit repeat. Either only the desired file should be downloaded, or nothing.

I tested again: the script downloaded only the latest installer (2148th). Then I waited a long time for the miraculous appearance of other files, but the miracle did not happen.

Are you sure you deleted the early test results before testing my script?

I’ve just re-loaded the script into a newly opened script editor (after rebooting the Mac) and ran it.

It runs and after the run has completed I have a new download. This time I got ‘emclient-v9.1.2125_Mac.pkg’.

I ran it again and it downloaded ‘emclient-v9.1.2109_Mac.pkg’

Therefore it always looks to download the installer that is the one previous to the last download it carried out.

Or maybe not, I ran it again and got ‘emclient-v9.1.2125_Mac.pkg’ again.

This is so weird.

If there anything I need to clear / or do before running the script?

It’s hard for me to understand the reason for the failure of the script on your computer, because everything works stably for me. I see 3 possible reasons:

  1. I have added the curl command to the Firewall exceptions for a long time. To avoid Apple Wishlist. That is, I allowed it to accept incoming connections.

  2. The speed of our computers can vary greatly. Try playing with the timeout value (2 sec, 10 sec).

  3. Try to remove the indication of error number 28. Use just on error. Because maybe we have different versions of curl command.

I tested again. The only downloaded file is emclient-v9.1.2148_Mac.pkg

Looks like the exit repeat is inside the error handler’s on error block, so it would only exit when there’s an error, right?

I played here with timeouts and as I see, the problem should be little timeout indicated in the script (that is, reason 2) of post #27). Also in some cases occurs error number 35 instead of number 28. So, it is better to not indicate on error number explicitly.

That is, stable script for slow computers (or, slow connections) should be like this:


repeat with aNumber from 2200 to 2109 by -1
	
	set MyDownloadPlace to quoted form of (POSIX path of (path to downloads folder) & "emclient-v9.1." & aNumber & "_Mac.pkg")
	set anURL to "https://cdn-dist.emclient.com/dist/v9.1." & aNumber & "_Mac/setup.pkg"
	
	try
		set maybe to do shell script "curl " & quoted form of anURL & " -m 10.0" -- or, bigger timeout 
	on error -- may occur error number 28 or 35
		do shell script "curl -L " & quoted form of anURL & " -o " & MyDownloadPlace
		exit repeat
	end try
	
end repeat

.
.

Throwing an error means that the script found a “rich” link, not a dummy link. Therefore, we immediately download its contents and leave the repeat loop.

I had removed the progress stuff because it worked so fast, but I put it back for this version, then I noticed that your version was using “the end of theURLs” rather than “the beginning of theURLs”. The difference means that the order is the reverse of what you wanted.

So this version is fast and has the progress tracking. I also combined the CURL routine @KniazidisR suggested.

use AppleScript version "2.4" -- Yosemite (10.10) or later
use scripting additions
use framework "Foundation"

set theURLS to {}
set totalSteps to 2200 - 2148
set progress total steps to totalSteps
set completedSteps to 0
set progress description to "2148 to 2200"

repeat with aNumber from 2148 to 2200

        --Note, using "the beginning" rather than "the end" puts them in the desired order
	set the beginning of theURLS to ("https://cdn-dist.emclient.com/dist/v9.1." & aNumber as text) & "_Mac/setup.pkg"
end repeat
set pagesFound to {}
set pagesNotFound to {}

set AppleScript's text item delimiters to {tab}

repeat with thisURL in theURLS
	set progress additional description to completedSteps & "of " & totalSteps
	set progress completed steps to completedSteps
	set {theURL, pageStatus} to WebPageHTML(thisURL as text)
	if pageStatus is "page found" then
		set the beginning of pagesFound to {"Found", theURL} as text
		exit repeat
	else
		set completedSteps to completedSteps + 1
		--set the end of pagesNotFound to {"Not found", theURL} as text
	end if
end repeat
set AppleScript's text item delimiters to {return & return}

--You could do this:

set MyDownloadfolder to (POSIX path of (path to downloads folder)) --& "emclient-v9.1." & 1 & "_Mac.pkg")
set MyDownloadedFile to quoted form of MyDownloadfolder & "emclient-v9.1." & 1 & "_Mac.pkg"
do shell script "curl -L " & quoted form of theURL & " -o " & MyDownloadedFile
tell application "Finder" to open (POSIX file MyDownloadfolder as alias)
 
on WebPageHTML(pageURLStr)
	set nsPageURL to current application's |NSURL|'s URLWithString:pageURLStr
	set {nsPageHTML, theError} to current application's NSData's dataWithContentsOfURL:nsPageURL options:0 |error|:(reference)
	if nsPageHTML = missing value then
		return {pageURLStr, "page not found"}
	else
		return {pageURLStr, "page found"}
	end if
end WebPageHTML

I tried this new combined script, it ran lightning fast and made a download, that download was entitled ‘emclient-v9.1.1_Mac.pkg’, so I’m not sure what version was actually downloaded?’

Update - I installed the version that was downloaded and it was version 9.1.2148, so the latest version, just not named correctly?

I just realised I think it’s the '& 1 ’ in the latter part of the script which needs changing out to the actual count variable, not sure how to do that with all the loops etc going on.

So, as often happens, once we can make a script work, we streamline it and make it more efficient. Try this:


use AppleScript version "2.4" -- Yosemite (10.10) or later
use scripting additions
use framework "Foundation"

set totalSteps to 2200 - 2148
set progress total steps to totalSteps
set completedSteps to 0
set progress description to "2200 to 2148"

repeat with aNumber from 2200 to 2148 by -1
	set thisURL to ("https://cdn-dist.emclient.com/dist/v9.1." & aNumber as text) & "_Mac/setup.pkg"
	set progress additional description to completedSteps & " of " & totalSteps
	set progress completed steps to completedSteps
	set {theURL, pageStatus} to WebPageHTML(thisURL as text)
	if pageStatus is "page found" then
		set progress completed steps to totalSteps
		
		exit repeat

	else
		set completedSteps to completedSteps + 1
	end if
end repeat

set MyDownloadfolder to (POSIX path of (path to downloads folder))
set MyDownloadedFile to quoted form of MyDownloadfolder & "emclient-v9.1." & (aNumber as text) & "_Mac.pkg"

do shell script "curl -L " & quoted form of theURL & " -o " & MyDownloadedFile

tell application "Finder" to open (POSIX file MyDownloadfolder as alias)

on WebPageHTML(pageURLStr)
	set nsPageURL to current application's |NSURL|'s URLWithString:pageURLStr
	set {nsPageHTML, theError} to current application's NSData's dataWithContentsOfURL:nsPageURL options:0 |error|:(reference)
	if nsPageHTML = missing value then
		return {pageURLStr, "page not found"}
	else
		return {pageURLStr, "page found"}
	end if
end WebPageHTML

It seems that today one of the lower numbers is the first available. But the script is still pretty fast.

And this is without all the progress stuff that I don’t think you need, since it’s so fast:

use AppleScript version "2.4" -- Yosemite (10.10) or later
use scripting additions
use framework "Foundation"

repeat with aNumber from 2200 to 2148 by -1
	set thisURL to ("https://cdn-dist.emclient.com/dist/v9.1." & aNumber as text) & "_Mac/setup.pkg"
	set {theURL, pageStatus} to WebPageHTML(thisURL as text)
	
	if pageStatus is "page found" then exit repeat
	
end repeat

set MyDownloadfolder to (POSIX path of (path to downloads folder))
set MyDownloadedFile to quoted form of MyDownloadfolder & "emclient-v9.1." & (aNumber as text) & "_Mac.pkg"
do shell script "curl -L " & quoted form of theURL & " -o " & MyDownloadedFile
tell application "Finder" to open (POSIX file MyDownloadfolder as alias)

on WebPageHTML(pageURLStr)
	set nsPageURL to current application's |NSURL|'s URLWithString:pageURLStr
	set {nsPageHTML, theError} to current application's NSData's dataWithContentsOfURL:nsPageURL options:0 |error|:(reference)
	if nsPageHTML = missing value then
		return {pageURLStr, "page not found"}
	else
		return {pageURLStr, "page found"}
	end if
end WebPageHTML

Totally and utterly awesome guys. This is exactly what I wanted to achieve. I’ll stick with the version with the graphics though as I do like to see the information displayed while it runs. I’ve commented out the line to open Finder too as I use Forklift and aways have that open to my download folder.

I’m so very well pleased with what has been achieved here.

@estockly:

With this line:

you are already downloading the file. Afterwards you load them again via curl.

With this you can only check if the file exists:


on WebPageHTML(pageURLStr)
	if word 3 of (do shell script "curl -s --head " & pageURLStr) is equal to "200" then
		return {pageURLStr, "page found"}
	else
		return {pageURLStr, "page not found"}
	end if
end WebPageHTML

Interestingly enough tonight I changed the values in the script to scan from 2200 to 2149, knowing that 2148 was the latest build.

It downloaded ‘emclient-v9.1.2150_Mac.pkg’ at 255 bytes in size.

On examining the file it turned out to be a text file and it contained:

Any ideas how to get round this one guys?

It may not be enough to check if data is returned for an incorrect URL. The HTTP status code (200) must be checked for a valid URL (see my previous post).

Edit: If the download is done outside the repeat loop, then you have to remember if the url was found (see “isFound” in the sample). Otherwise the download is done with the last number.

This works even if the web server returns an error page.
(based on code from estockly)


use AppleScript version "2.4" -- Yosemite (10.10) or later
use scripting additions

set isFound to false

repeat with aNumber from 2200 to 2148 by -1
	set theUrl to ("https://cdn-dist.emclient.com/dist/v9.1." & aNumber as text) & "_Mac/setup.pkg"
	if WebPageExist(theUrl) then
		set isFound to true
		set MyDownloadfolder to (POSIX path of (path to downloads folder))
		set MyDownloadedFile to MyDownloadfolder & "emclient-v9.1." & (aNumber as text) & "_Mac.pkg"
		do shell script "curl -L " & theUrl & " -o " & quoted form of MyDownloadedFile
		open location "file://" & MyDownloadedFile
		exit repeat
	else
		delay 0.01
	end if
end repeat

if not isFound then
	display alert "Download eM Client" message "Release not found." as critical giving up after 10
end if

on WebPageExist(pageURLStr)
	if word 3 of (do shell script "curl -s --head " & pageURLStr) is equal to "200" then
		return true
	else
		return false
	end if
end WebPageExist

Okay I just tried the new script and that downloaded version 2148 but then launched the installer as soon as it was downloaded. Ideally the installer should not auto launch.

I then changed the script to set 2149 as the bottom limit and ran it again. That time I got the expected ‘Release not found.’ dialog :slight_smile:

Delete or comment out (–) this line:


 open location "file://" & MyDownloadedFile