Monday, March 27, 2017

#1 2017-03-08 04:48:29 am

Eastland
Member
Registered: 2015-01-13
Posts: 16

Web Image Download from CSV file.

Hi,

I was hoping somebody would able to help with a script that does the following

Scans a .csv file column A

Adds each cell in turn to the end of a url : http://i1.adis.ws/i/jpl/contents_of_cell_A1

Accesses the site 

Then downloads the image to a specified local folder.

Repeats until the last cell.

I am currently processing this manually and attempting to create something in automator but not having much luck.

If anyone could help that would be awesome

Cheers big_smile

Offline

 

#2 2017-03-08 02:07:22 pm

t.spoon
Member
From: BFE, Massachusetts
Registered: 2013-01-12
Posts: 98

Re: Web Image Download from CSV file.

This worked on a test .csv I made with just image URL's in the first column:

Applescript:

use AppleScript version "2.4" -- Yosemite (10.10) or later
use scripting additions

set theSpreadsheet to choose file with prompt "Please select your csv file with the image URL's" of type {"csv"} without multiple selections allowed

tell application "Finder"
   set downloadsFolder to path to downloads folder
   set saveFolder to ((downloadsFolder as text) & "Script Downloaded Images:")
   if not (exists saveFolder) then make new folder at downloadsFolder with properties {name:"Script Downloaded Images"}
   set saveFolder to POSIX path of saveFolder
end tell

set spreadsheetData to read theSpreadsheet
set dataLines to the paragraphs of spreadsheetData
set theURLs to {}
repeat with aLine in dataLines
   set delimitHolder to AppleScript's text item delimiters
   set AppleScript's text item delimiters to ","
   copy text item 1 of aLine to the end of theURLs
   -- may want to insert validation in case it doesn't start with "HTTP" or end with the right extension, etc.
   set AppleScript's text item delimiters to delimitHolder
end repeat

set imageCount to the count of theURLs

set missingImages to {}
set downloadedImages to {}
repeat with imageURL in theURLs
   set delimitHolder to AppleScript's text item delimiters
   set AppleScript's text item delimiters to "/"
   set imageName to the last text item of imageURL
   set AppleScript's text item delimiters to delimitHolder
   set downloadFilePath to saveFolder & imageName as text
   
   set shellOutput to (do shell script "
           
           image=\""
& (contents of imageURL as text) & "\"
           response=$(curl --write-out %{http_code} --silent --output /dev/null $image)
           
           if [ \"$response\" == \"200\" ]; then
           curl $image -o \""
& downloadFilePath & "\"
           echo \"Image downloaded\"
           else
           echo \"Image does not exist\"
           fi
           
           "
)
   
   if shellOutput is "Image does not exist" then
       copy imageURL to the end of missingImages
   else
       copy imageURL to the end of downloadedImages
   end if
end repeat

set missingImageDialogText to ""
if (count of missingImages) > 0 then
   set missingImageDialogText to "The following images failed to download:"
   repeat with anImage in missingImages
       set missingImageDialogText to missingImageDialogText & return & anImage
   end repeat
end if
display dialog "The script completed." & return & (count of downloadedImages) & " out of " & imageCount & " images were successfully downloaded to: " & saveFolder & return & missingImageDialogText buttons {"Cancel", "OK"} default button "OK"

Offline

 

#3 2017-03-08 04:51:24 pm

ccstone
Member
Registered: 2009-02-07
Posts: 403

Re: Web Image Download from CSV file.

Hey There,

It's pretty simple if you know how.

This script will run as is and will download 3 images to a new dated folder in your Downloads folder.

Applescript:


-------------------------------------------------------------------------------------------
# Auth: Christopher Stone
# dCre: 2017/03/08 16:00
# dMod: 2017/03/08 16:28
# Appl: AppleScript & the Shell
# Task: Download image URLs from a CSV file to a dated folder in the ~/Downloads folder.
# Libs: None
# Osax: None
# Tags: @Applescript, @Script, @Download, @Image, @URLs, @CSV, @Dated, @Folder
-------------------------------------------------------------------------------------------

set shCMD to "
export PATH=/opt/local/bin:/opt/local/sbin:/usr/local/bin:$PATH;

baseURL='http://members.aceweb.com/randsautos/photogallery/ferrari/enzo/'

# Simulate reading the csv data:
csvData='
Ferrari-Enzo-001.jpg,Line01-Field02,Line01-Field03
Ferrari-Enzo-002.jpg,Line02-Field02,Line02-Field03
Ferrari-Enzo-003.jpg,Line03-Field02,Line03-Field03
'

urlList=$(sed -En '/[[:alnum:]]+/{
   s!,.+!!
   s!^!'\"$baseURL\"'!
   s!^!\"!g
   s!$!\"!g
   s!^!url = !g
   p
}' <<< \"$csvData\")

newFolderPath=~/Downloads/'Downloaded Files - '$(date \"+%Y-%m-%d %H.%M.%S\")
mkdir -p \"$newFolderPath\"

cd \"$newFolderPath\"

echo \"$urlList\" \\
| curl -Ls --remote-name-all -L --user-agent 'Opera/9.70 (Linux ppc64 ; U; en) Presto/2.2.1' -K -
"


do shell script shCMD

-------------------------------------------------------------------------------------------

You can read the file with AppleScript, or more simply by using the cat command in the shell.

cat '/path/to/your/file.csv'

Make sure to quote the path if it has any spaces in it.

So – in the script above you'd replace the “csvData=” segment with:

csvData=$(cat '/path/to/your/file.csv')

--
Chris
··················································································
{ MacBookPro6,1 · 2.66 GHz Intel Core i7 · 8GB RAM · OSX 10.12.3 }

Offline

 

#4 2017-03-08 04:54:36 pm

ccstone
Member
Registered: 2009-02-07
Posts: 403

Re: Web Image Download from CSV file.

Okay, let's make that simpler for you and change it to read a file.

Given a file named “URL LIST.csv” in your Downloads folder (see dataFilePath in the script) with content of:

Ferrari-Enzo-001.jpg,Line01-Field02,Line01-Field03
Ferrari-Enzo-002.jpg,Line02-Field02,Line02-Field03
Ferrari-Enzo-003.jpg,Line03-Field02,Line03-Field03

The script will read the file directly into ‘sed’ to isolate the file names and build the URLs.

Applescript:


-------------------------------------------------------------------------------------------
# Auth: Christopher Stone
# dCre: 2017/03/08 16:00
# dMod: 2017/03/08 16:50
# Appl: Miscellaneous
# Task: Download image URLs from a CSV file to a dated folder in your ~/Downloads folder.
# Libs: None
# Osax: None
# Tags: @Applescript, @Script, @Download, @Image, @URLs, @CSV, @Dated, @Folder, @curl, @sed
-------------------------------------------------------------------------------------------

set shCMD to "
export PATH=/opt/local/bin:/opt/local/sbin:/usr/local/bin:$PATH;

# Note the single-quotes in the path due to spaces.
dataFilePath=~/'Downloads/URL LIST.csv'

baseURL='http://members.aceweb.com/randsautos/photogallery/ferrari/enzo/'

urlList=$(sed -En '/[[:alnum:]]+/{
   s!,.+!!
   s!^!'\"$baseURL\"'!
   s!^!\"!g
   s!$!\"!g
   s!^!url = !g
   p
}' \"$dataFilePath\" )

newFolderPath=~/Downloads/'Downloaded Files - '$(date \"+%Y-%m-%d %H.%M.%S\")
mkdir -p \"$newFolderPath\"

cd \"$newFolderPath\"

echo \"$urlList\" | curl -Ls --remote-name-all -L --user-agent 'Opera/9.70 (Linux ppc64 ; U; en) Presto/2.2.1' -K -
"

do shell script shCMD

-------------------------------------------------------------------------------------------

-Chris

Offline

 

#5 2017-03-09 08:34:40 am

t.spoon
Member
From: BFE, Massachusetts
Registered: 2013-01-12
Posts: 98

Re: Web Image Download from CSV file.

Whoops, I see I missed that the csv doesn't contain the entire URL, that it's just the end of it.

Amended script here:

Applescript:

use AppleScript version "2.4" -- Yosemite (10.10) or later
use scripting additions

set baseURL to "http://i1.adis.ws/i/jpl/"

set theSpreadsheet to choose file with prompt "Please select your csv file with the image URL's" of type {"csv"} without multiple selections allowed

tell application "Finder"
   set downloadsFolder to path to downloads folder
   set saveFolder to ((downloadsFolder as text) & "Script Downloaded Images:")
   if not (exists saveFolder) then make new folder at downloadsFolder with properties {name:"Script Downloaded Images"}
   set saveFolder to POSIX path of saveFolder
end tell

set spreadsheetData to read theSpreadsheet
set dataLines to the paragraphs of spreadsheetData
set theURLs to {}
repeat with aLine in dataLines
   set delimitHolder to AppleScript's text item delimiters
   set AppleScript's text item delimiters to ","
   set newURL to baseURL & text item 1 of aLine
   copy newURL to the end of theURLs
   set AppleScript's text item delimiters to delimitHolder
end repeat

set imageCount to the count of theURLs

set missingImages to {}
set downloadedImages to {}
repeat with imageURL in theURLs
   set delimitHolder to AppleScript's text item delimiters
   set AppleScript's text item delimiters to "/"
   set imageName to the last text item of imageURL
   set AppleScript's text item delimiters to delimitHolder
   set downloadFilePath to saveFolder & imageName as text
   
   set shellOutput to (do shell script "
           
           image=\""
& (contents of imageURL as text) & "\"
           response=$(curl --write-out %{http_code} --silent --output /dev/null $image)
           
           if [ \"$response\" == \"200\" ]; then
           curl $image -o \""
& downloadFilePath & "\"
           echo \"Image downloaded\"
           else
           echo \"Image does not exist\"
           fi
           
           "
)
   
   if shellOutput is "Image does not exist" then
       copy imageURL to the end of missingImages
   else
       copy imageURL to the end of downloadedImages
   end if
end repeat

set missingImageDialogText to ""
if (count of missingImages) > 0 then
   set missingImageDialogText to "The following images failed to download:"
   repeat with anImage in missingImages
       set missingImageDialogText to missingImageDialogText & return & anImage
   end repeat
end if
display dialog "The script completed." & return & (count of downloadedImages) & " out of " & imageCount & " images were successfully downloaded to: " & saveFolder & return & missingImageDialogText buttons {"Cancel", "OK"} default button "OK"

Offline

 

#6 2017-03-10 04:27:53 am

Eastland
Member
Registered: 2015-01-13
Posts: 16

Re: Web Image Download from CSV file.

Thank you so much for the help this works an absolute treat big_smile

Offline

 

#7 2017-03-10 10:33:54 am

t.spoon
Member
From: BFE, Massachusetts
Registered: 2013-01-12
Posts: 98

Re: Web Image Download from CSV file.

Just out of curiosity, were you using my script, or ccstone's?

I think they're pretty similar except he's doing everything in the shell and I'm using as much Applescript as possible and just using the shell to call curl to do the actual download.

If you know some Applescript but not much shell, then mine might have some modification/maintainability advantage for you... of course, if you know shell better then Applescript, then the opposite would be true.

Offline

 

Board footer

Powered by FluxBB

[ Generated in 0.058 seconds, 10 queries executed ]

RSS (new topics) RSS (active topics)