Working with Folder containing a lot of files

Hi,

so I’m writing this script which does multiple things. One of them if copy a folder to a new location and once the copy finished, look in the new folder to get all files of the same type (.mhl) to run a terminla command on them. The script seems to work just fine on small folder, but when I do the test with a real production folder I seem to get some strange behavior. Since this folder contains a lot of files (around 2000) and is pretty heavy (40 gig) I put a timeout of 5000000 seconds to make sure that Finder doesn’t time out during copy. Which works fine, the whole folder copy fully but once the copy is done, the finder seem to ‘‘stop responding’’ therefore, the terminal cmd is never launch. My guess is that the finder is having trouble searching folder and subfolders containing a huge number of files (as if I delet the part where I tell the Finder to search for mlh, everything seems to work fine). Any way I can
workaround this issue? Here’s the part of the script that his causing the issue.

thanks a lot
-F

tell application "Finder"
	set sourceFile to choose folder with prompt "Choose a folder to copy:"
	set targetFolder to choose folder with prompt "And where do you want it to go?"
end tell
with timeout of 5000000 seconds
	tell application "Finder"
		try
			set new_folder to (duplicate sourceFile to targetFolder) as alias
		on error errMsg number errNum
			display dialog "Error: " & errNum & return & errMsg
		end try
		update targetFolder
	end tell
end timeout
tell application "Finder" to set theFiles to (files of entire contents of folder new_folder whose name contains ".mhl") as alias list
tell application "Terminal"
	if not (exists window 1) then reopen
	activate
end tell
repeat with aFile in theFiles
	set filePath to aFile's POSIX path
	tell application "Terminal"
		do script "mhl verify -f " & quoted form of filePath
	end tell
end repeat

Hi,

I recommend to use the shell for the duplicate and find task.
It’s doesn’t block the Finder and is much much faster

set sourceFolder to POSIX path of (choose folder with prompt "Choose a folder to copy:")
set targetFolder to POSIX path of (choose folder with prompt "And where do you want it to go?")
set {TID, text item delimiters} to {text item delimiters, "/"}
set sourceFolderName to text item -2 of sourceFolder
set text item delimiters to TID
do shell script "/usr/bin/rsync -vautE " & quoted form of text 1 thru -2 of sourceFolder & space & quoted form of targetFolder
set theFiles to do shell script "/usr/bin/find " & quoted form of (targetFolder & sourceFolderName) & " -type f -name '*mhl'"
tell application "Terminal"
	if not (exists window 1) then reopen
	activate
end tell
repeat with aFile in theFiles
	tell application "Terminal"
		do script "mhl verify -f " & quoted form of aFile
	end tell
end repeat

if it’s not required to keep the terminal window open you could do this, which is still faster

set sourceFolder to POSIX path of (choose folder with prompt "Choose a folder to copy:")
set targetFolder to POSIX path of (choose folder with prompt "And where do you want it to go?")
set {TID, text item delimiters} to {text item delimiters, "/"}
set sourceFolderName to text item -2 of sourceFolder
set text item delimiters to TID
do shell script "/usr/bin/rsync -vautE " & quoted form of text 1 thru -2 of sourceFolder & space & quoted form of targetFolder
do shell script "/usr/bin/find " & quoted form of (targetFolder & sourceFolderName) & " -type f -name '*mhl' -exec mhl verify -f {} \\;"

Hi Stefan,

thanks a lot for the reply. I did some test and it indeed solves the timeout issue that was occurring after the copy. But there seems to be a problem on the detection of .mhl files. Once the copy finish, instead of looking for the .mh file (only had one present for the test) and run the terminal command “mhl verify -f” to it, I have one terminal window opening for each files (even the ones that are not .mhl) so like a 100 terminal windows opening at the same time. It seems like if it was not searching for .mhl files but instead, try to run the cmd line to each file of the folder.

Also, it seems to look for everything that is present in the destination folder. The thing is (and I forgot to mention it) This destination folder will be incremental, I’m gonna duplicate one folder every day to the destination so the destination folder will have more than one subfolder in it (i.e Day01, Day 02, Day 03…).

So when I’m gonna duplicate Day 04, I just want to run mhl verification on this folder as all of the others will have been done prior.

thanks

Actually, I don’t think it’s trying to open any file in the terminal. It just open a lot of terminal windows with the wrong path to source, going mhl verify -f ‘m’ and changing the ‘‘letter’’ every time it open a new terminal window (having mhl verify -f ‘m’ ; mhl verify -f ‘i’ ; mhl verify -f ‘h’ ; etc…)

fcengarle$ mhl verify -f ‘m’
Error occured during parsing of MHL file ‘/Users/fcengarle/m’.
Description: MHL file is not found.
voutedev04:~ fcengarle$

sorry, I forgot to make a list from the result of the find line.
This script searches only 1 level deep

set sourceFolder to POSIX path of (choose folder with prompt "Choose a folder to copy:")
set targetFolder to POSIX path of (choose folder with prompt "And where do you want it to go?")
set {TID, text item delimiters} to {text item delimiters, "/"}
set sourceFolderName to text item -2 of sourceFolder
set text item delimiters to TID
do shell script "/usr/bin/rsync -vautE " & quoted form of text 1 thru -2 of sourceFolder & space & quoted form of targetFolder
set theFiles to paragraphs of (do shell script "/usr/bin/find " & quoted form of (targetFolder & sourceFolderName) & " -mindepth 1 -maxdepth 1 -type f -name '*.mhl")
tell application "Terminal"
	if not (exists window 1) then reopen
	activate
end tell
repeat with aFile in theFiles
	tell application "Terminal"
		do script "mhl verify -f " & quoted form of aFile
	end tell
end repeat

Ok, now the terminal isn’t going crazy But I get an error message on the find part saying

‘‘error "sh: -c: line 0: unexpected EOF while looking for matching `’’
sh: -c: line 1: syntax error: unexpected end of file" number 2’’

Also, is it possible to search more than one level deep as those will be camera folders so their architecture will look like this

Folder/subfolder/subfolder/mhl file

thanks again for the helping hand

there is a single quote missing

. -name '*.mhl'")

I’m a bit confused about

Awesome!

Ok, sorry about the confusion. What I meant is that I’m gonna drop folders everyday. Here’s an example

On Day one, I will drop 2 folders Day01A and Day01B
On Day two, I will duplicate 2 other folders Day02A and Day02B

Now, what I want is for the find to go in subfolders as the architecture will look like Destination/Day01A/ARRIRAW_files/A177R10O/mhl file

What I don’t want is the find to look on Day01 folders once the copy of Day02 finishes (as Day01 mhl files will already had been verify the day before). I don’t want to run verification on mho files that were done the day before.

I just removed mindepth 1 -maxdepth 1 and it’s working like a charm

Now can I just bug you one more time and ask if there’s a way to see the progress of the copy once it’s running? As for now, we only see the applescript running but don’t know how much time’s left for the copy to finish.

As you have an open Terminal window anyway, try this, it displays everything in a single window

set sourceFolder to POSIX path of (choose folder with prompt "Choose a folder to copy:")
set targetFolder to POSIX path of (choose folder with prompt "And where do you want it to go?")
set {TID, text item delimiters} to {text item delimiters, "/"}
set sourceFolderName to text item -2 of sourceFolder
set text item delimiters to TID
tell application "Terminal"
	if not (exists window 1) then reopen
	activate
	do script "/usr/bin/rsync -vautE " & quoted form of text 1 thru -2 of sourceFolder & space & quoted form of targetFolder in window 1
end tell
set theFiles to paragraphs of (do shell script "/usr/bin/find " & quoted form of (targetFolder & sourceFolderName) & " -type f -name '*.mhl'")
tell application "Terminal"
	repeat with aFile in theFiles
		do script "mhl verify -f " & quoted form of aFile
	end repeat
end tell

That’s perfect!

Also, as I mentioned, my script is doing a lot of studs. One of them is sending an email once everything’s complete. It use to work fine before, but now that I’m using shell script, the mail doesn’t seem to wait for the terminal to complete his mhl verify before it sends the message. Instead, it send it as soon as the terminal start its mhl verification. Is there a way that we can tell to wait for the terminal to finish before sending the mail?

here’s my complete script

set sourceFolder to POSIX path of (choose folder with prompt "Choose a folder to copy:")
set targetFolder to POSIX path of (choose folder with prompt "And where do you want it to go?")
set {TID, text item delimiters} to {text item delimiters, "/"}
set sourceFolderName to text item -2 of sourceFolder
set text item delimiters to TID

property defaultGroups : {"Family", "Friends"}
property defaultLabels : {}

-- Prompt for the group to use
tell application "Contacts" to set everyGroup to name of every group
if (count of everyGroup) is greater than 0 then
	set theGroups to choose from list everyGroup with prompt ¬
		"Send this message to which group?" default items defaultGroups with multiple selections allowed
end if

-- Prompt for message subject
set theResult to display dialog "What would you like the subject of the message to be?" default answer ""
set theSubject to text returned of theResult

-- Prompt for whether an attachment is desired. If so, prompt for the location of the file.
set theResult to display dialog "Would you like to attach some files to this message?" buttons {"Yes", "No"} default button 1
set wantsAttachment to button returned of theResult
if wantsAttachment is equal to "Yes" then
	set theAttachment to choose file
end if

-- Prompt for message body
set theResult to display dialog "What would you like to say in the body of the message?" default answer ""
set theBody to text returned of theResult

-- Go through each account and constuct a list of possible addresses
-- to use as a return address for this message.
tell application "Mail"
	set listOfSenders to {}
	set everyAccount to every account
	repeat with eachAccount in everyAccount
		set everyEmailAddress to email addresses of eachAccount
		if (everyEmailAddress is not equal to missing value) then
			repeat with eachEmailAddress in everyEmailAddress
				set listOfSenders to listOfSenders & {(full name of eachAccount & " <") as string}
			end repeat
		end if
	end repeat
end tell

-- Prompt the user to select which account to send this message from.
set theResult to choose from list listOfSenders with prompt "Which account would you like to send this message from?" without multiple selections allowed
if theResult is not equal to false then
	set theSender to item 1 of theResult
	do shell script "/usr/bin/rsync -vautE " & quoted form of text 1 thru -2 of sourceFolder & space & quoted form of targetFolder
	set theFiles to paragraphs of (do shell script "/usr/bin/find " & quoted form of (targetFolder & sourceFolderName) & " -type f -name '*.mhl'")
	tell application "Terminal"
		if not (exists window 1) then reopen
		activate
	end tell
	repeat with aFile in theFiles
		tell application "Terminal"
			do script "mhl verify -f " & quoted form of aFile
		end tell
	end repeat
	tell application "Mail"
		
		-- Properties can be specified in a record when creating the message or
		-- afterwards by setting individual property values.
		set newMessage to make new outgoing message with properties {subject:theSubject, content:theBody & return & return}
		tell newMessage
			-- Default is false. Determines whether the compose window will
			-- show on the screen or whether it will happen in the background./3693.
			set visible to true
			set sender to theSender
			set theEmails to {}
			tell application "Contacts"
				repeat with aGroup in theGroups
					repeat with aPerson in people of group aGroup
						repeat with anEmail in emails of aPerson
							if (label of anEmail is in defaultLabels) or defaultLabels is {} then
								if theEmails does not contain the value of anEmail then
									copy (value of anEmail) to end of theEmails
								end if
							end if
						end repeat
					end repeat
				end repeat
			end tell
			repeat with anEmail in theEmails
				make new to recipient at end of to recipients with properties {address:anEmail}
			end repeat
			tell content
				if (wantsAttachment is equal to "Yes") then
					-- Position must be specified for attachments
					make new attachment with properties {file name:theAttachment} at after the last paragraph
				end if
			end tell
		end tell
		
		-- Bring the new compose window to the foreground, in all its glory
		activate
		send newMessage
	end tell
end if