Saturday, June 24, 2017

#1 2017-04-20 02:56:40 pm

pcLoadLetter
Member
Registered: 2016-09-23
Posts: 26

Making Filename Searches Lightening Fast

This script searches through several different directories (and their subdirectories). The problem is that I'm searching through so many different files in the last 2 directories, which makes the shellscript far slower. (I went from the search taking seconds per image to it taking up to 5 to 10 minutes for an image that isn't found in the smaller directories. to several hours when I added those two directories to the script. This script loops through hundreds or even a thousand plus files, so I need to make it faster to work with our workflow).

What can I do to make automating searching through large directories faster? I'm open to using a different shell script command or language. Any advice is greatly appreciated. Thanks!


Applescript:

--Stringing together the missing image to search for
set search_name to "*" & blain_number & "*" as text


--Specifying the path to the user's hard drive
set my_HD to path to startup disk as string

--Searching for the images
tell application "Finder"
set image_search1 to paragraphs of (do shell script "find " & quoted form of (POSIX path of images_folder) & " -name '" & search_name & "' -type f") -->searching
if (image_search1 as string) is "" then
set image_search1 to paragraphs of (do shell script "find " & quoted form of (POSIX path of processing_folder) & " -name '" & search_name & "' -type f") -->searching
end if
if (image_search1 as string) is "" then
set image_search1 to paragraphs of (do shell script "find " & quoted form of (POSIX path of r_drive_folder) & " -name '" & search_name & "' -type f") -->searching
end if
if (image_search1 as string) is "" then
set image_search1 to paragraphs of (do shell script "find " & quoted form of (POSIX path of adv_print_folder) & " -name '" & search_name & "' -type f") -->searching
end if
end tell


--Adding the image to the found link list
if (image_search1 as string) is not "" then
set found_image_list to found_image_list & image_search1
set found_number_list to found_number_list & blain_number
else
set missing_link_list to missing_link_list & blain_number
end if

Offline

 

#2 2017-04-20 04:57:24 pm

Marc Anthony
Member
From: Dallas, TX
Registered: 2006-04-27
Posts: 749

Re: Making Filename Searches Lightening Fast

Hi. There is little you can do to speed up the shell script except limit the traversal's depth, as searching through jillions of files and folders recursively takes time. You do need to nix the Finder call; it has nothing to do with the shell.

Offline

 

#3 2017-04-20 06:28:50 pm

pcLoadLetter
Member
Registered: 2016-09-23
Posts: 26

Re: Making Filename Searches Lightening Fast

What about trying something other than shellscript? Bridge seems to find the files by hand faster and it likes javascript. Maybe there's something in JavaScript or another language? I think searching through less files would be a great idea! The hard part is convincing other people to implement a better organized image library. The code is starting to seem easy by comparison. lol

Last edited by pcLoadLetter (2017-04-20 06:45:06 pm)

Offline

 

#4 2017-04-20 06:53:40 pm

Shane Stanley
Member
From: Australia
Registered: 2002-12-07
Posts: 5022

Re: Making Filename Searches Lightening Fast

What Marc says. And if you're searching on remote volumes, which it sounds like, doubly so. You can speed up searches of remote volumes by caching the contents locally, but that can work only when the contents are not changing too often, and it also complicates things.


Shane Stanley <sstanley@myriad-com.com.au>
www.macosxautomation.com/applescript/apps/

Offline

 

#5 2017-04-24 04:37:58 pm

t.spoon
Member
From: BFE, Massachusetts
Registered: 2013-01-12
Posts: 185

Re: Making Filename Searches Lightening Fast

Spotlight may offer a speed improvement over shell "find." There is some talk of remote volumes, in which case they might not be Spotlight indexed. But if they're local, or remote volumes hosted on a Mac and shared via AFP supporting remote Spotlight queries, then I'd expect using Spotlight instead of UNIX "Find" to be a lot faster. Examples of Applescript & Spotlight in this thread:

http://macscripter.net/viewtopic.php?id=27833

Another possible alternative for a remote volume shared by AFP that's not Spotlight Searchable is to script Find Any File. http://apps.tempel.org/FindAnyFile/ There's no Applescript dictionary, but it looks pretty amenable to UI scripting.

But if it's a remote SMB volume, then neither of these will be of any help speed-wise.

You mentioned Bridge is faster - Bridge builds its own index like spotlight. If Bridge works in your scenario, then yes, it is scriptable with Javascript. And if you want to stick with everything but the search being in Applescript, Bridge has one Applescript command: "do javascript."

Another option, if it is a remote volume shared by something other than AFP and you are OK with allowing the overhead of a local index of a remote volume, is to use Quicksilver to index the required parts of the remote volume, then Applescript a Quicksilver search. https://qsapp.com/ That's as fast as spotlight when searching and works with remote SMB/NFS/webdav volumes. Quicksilver has a good Applescript Dictionary.

If this is confusing, let me know how this volume is accessed (local or remote and, if remote, the protocol), and whether or not you have Bridge indexing it anyway, and I can help you zoom in on possibly strategies.


Hackintosh built February, 2012 |  Mac OS Sierra
GIGABYTE GA-Z68X-UD3H-B3 | Core i5 2500k | 16 GB DDR3 | GIGABYTE Geforce 1050 TI 4GB
250 GB Samsung 850 EVO | 4 TB RAID
Dell Ultrasharp U3011 | Dell Ultrasharp 2007FPb

Offline

 

#6 2017-04-25 10:47:52 am

pcLoadLetter
Member
Registered: 2016-09-23
Posts: 26

Re: Making Filename Searches Lightening Fast

Thanks! I like the Bridge idea. I was having trouble getting the javascript right for the search. Here's what I found in the javascript guide and javascript reference: Any help would be greatly appreciated. (I had it search my desktop to test it, but it will end up searching a server connected by smb).

Applescript:

set my_folder to "/Users/user_name/Desktop/"
set my_search_criteria to "612369"

tell application "Adobe Bridge CS5.1"
   activate
   
   do javascript "
               #target 'bridge'
//Declaring the variables for the folder and search criteria
var search_folder = ['' & my_folder & ''];
var search_criteria = ['' & my_search_criteria & ''];

new SearchCriteria (searchField, operandType,
*searchFieldDisplay, *operands);

new SearchDefinition (criteriaList, defaultResultsLimit,
*ranks, *scopeSpecifiers);

new SearchSpecification (conditionList, conjunction, maximumResults,
rankOrdering, rankField, scopeSpecifiers);

obj.getBridgeURIForSearch
(scope, specification)

obj.getSearchDetails ()

/**/"

end tell

Last edited by pcLoadLetter (2017-04-25 10:49:45 am)

Offline

 

#7 2017-04-25 12:39:40 pm

t.spoon
Member
From: BFE, Massachusetts
Registered: 2013-01-12
Posts: 185

Re: Making Filename Searches Lightening Fast

It may be time to move to a javascript forum. Javascripting Bridge may be the best approach in your scenario, but you're on an Applescript forum... you may want to try the Adobe Bridge scripting forums:
https://forums.adobe.com/community/brid … _scripting
or Stack Overflow:
http://stackoverflow.com/

Maybe someone here also Javascripts and will chime in with help, but I wouldn't expect it on an Applescript forum.

Best of luck with it though,

Tom.


Hackintosh built February, 2012 |  Mac OS Sierra
GIGABYTE GA-Z68X-UD3H-B3 | Core i5 2500k | 16 GB DDR3 | GIGABYTE Geforce 1050 TI 4GB
250 GB Samsung 850 EVO | 4 TB RAID
Dell Ultrasharp U3011 | Dell Ultrasharp 2007FPb

Offline

 

#8 2017-04-25 06:26:16 pm

JMichaelTX
Member
From: Houston, TX (The Woodlands)
Registered: 2014-07-12
Posts: 133

Re: Making Filename Searches Lightening Fast

pcLoadLetter wrote:

This script searches through several different directories (and their subdirectories). The problem is that I'm searching through so many different files in the last 2 directories, which makes the shellscript far slower. (I went from the search taking seconds per image to it taking up to 5 to 10 minutes for an image that isn't found in the smaller directories. to several hours when I added those two directories to the script. This script loops through hundreds or even a thousand plus files, so I need to make it faster to work with our workflow).

What can I do to make automating searching through large directories faster? I'm open to using a different shell script command or language. Any advice is greatly appreciated. Thanks!

I'm curious.  Have you tried a manual Spotlight search, just using the part of the file name you are interested in?
If you only want image files, then you could limit the search by using "kind:image" (without the quotes).
Spotlight finds files near instantaneously for me on an iMac-27 with 2TB Fusion drive AND a 3TB LaCie ThunderBolt2 drive.

If Spotlight is quick enough for you, you can use the `mdfind` command in a shell script.
See:
mdfind Man Page - macOS - SS64.com
Spotlight syntax, mdfind examples, and metadata attributes

There are also some ASObjC searches that are very fast, but @ShaneStanley is better equipped to talk about those than I am.


iMac-27 Late 2015 Retina 5K Screen (& others)
macOS 10.11.6 (El Capitan)

Offline

 

#9 2017-05-01 11:24:20 am

pcLoadLetter
Member
Registered: 2016-09-23
Posts: 26

Re: Making Filename Searches Lightening Fast

I was experimenting with mdfind in shellscript. The one issue I had is that mdfind only searched through one subfolder in the specified folder. If, for instance, I had a folder named "AG1 Images" inside the specified "AG-high res images" folder, mdfind would find the image. If I put the image inside a second subfolder named "testing" within the "AG1 Images" folder within the specified "AG-high res images" folder, the script was unable to find the image. (There will be a lot of subfolders inside the folder that I need to search through.)

Is there something I can change with mdfind to get it to search through all sub-subfolders within that folder?


Applescript:

set search_folder to "/Volumes/NAS/Advertising Department/2_IMAGES - HIGH RES FOR PRINT/1_4c IMAGES/AG-high res images/"
set found_images to paragraphs of (do shell script "mdfind -onlyin " & quoted form of search_folder & " 'kMDItemFSName = \"*355390*\"'")

Offline

 

#10 2017-05-01 04:53:22 pm

pcLoadLetter
Member
Registered: 2016-09-23
Posts: 26

Re: Making Filename Searches Lightening Fast

How do I pass the information from Javascript back to AppleScript? The search is working from Bridge. I just need to make the results accessible from AppleScript

Applescript:

set high_res_folder to "/Volumes/sblain$/*SIGNS FOR PREPRESS/Agriculture Dept/Chicken Coops/high res images/"

tell application "Adobe Bridge CC 2017"
   activate
   
   do javascript "
   #target bridge;
   
   var search_folder = ['"
& high_res_folder & "'];
   alert(search_criteria);
var my_files = Folder(search_folder).getFiles('*811951*');
alert(my_files);
"

   
end tell

Offline

 

#11 2017-05-02 08:46:09 am

DJ Bazzie Wazzie
Member
From: the Netherlands
Registered: 2004-10-20
Posts: 2648

Re: Making Filename Searches Lightening Fast

pcLoadLetter wrote:

Is there something I can change with mdfind to get it to search through all sub-subfolders within that folder?

AppleScript Toolbox does that by default.

Applescript:

AST query metadata "kMDItemFSName == 'yes'" only in folders "/usr"

Offline

 

#12 2017-05-02 10:06:16 am

t.spoon
Member
From: BFE, Massachusetts
Registered: 2013-01-12
Posts: 185

Re: Making Filename Searches Lightening Fast

pcLoadLetter wrote:

How do I pass the information from Javascript back to AppleScript? The search is working from Bridge. I just need to make the results accessible from AppleScript

Applescript:

set high_res_folder to "/Volumes/sblain$/*SIGNS FOR PREPRESS/Agriculture Dept/Chicken Coops/high res images/"

tell application "Adobe Bridge CC 2017"
   activate
   
   do javascript "
   #target bridge;
   
   var search_folder = ['"
& high_res_folder & "'];
   alert(search_criteria);
var my_files = Folder(search_folder).getFiles('*811951*');
alert(my_files);
"

   
end tell

Javascript is similar to Applescript. They're both Smalltalk descendants. I think Javascript will return whatever variable was last returned in the program. And whatever it returns at the end, it returns to Applescript.

In this case, it looks like what you want is "my_files". I don't have Bridge to test, but I think you should just be able to make the last line of the program return "my_files" and call the javascript with

Applescript:

set myFiles to do javascript"

- Tom.


Hackintosh built February, 2012 |  Mac OS Sierra
GIGABYTE GA-Z68X-UD3H-B3 | Core i5 2500k | 16 GB DDR3 | GIGABYTE Geforce 1050 TI 4GB
250 GB Samsung 850 EVO | 4 TB RAID
Dell Ultrasharp U3011 | Dell Ultrasharp 2007FPb

Offline

 

#13 2017-05-03 07:14:38 am

DJ Bazzie Wazzie
Member
From: the Netherlands
Registered: 2004-10-20
Posts: 2648

Re: Making Filename Searches Lightening Fast

t.spoon wrote:

They're both Smalltalk descendants.

Can you be more specific? Because you could say that almost every modern programming languages is an descendant of Simula even if they are completely different. What do they have in common, I'm only curious. AFAIK JavaScript is a dialect from ECMAScript while AppleScript is an descendant from HyperTalk (syntax) and HyperCard (VM).

Offline

 

#14 2017-05-03 08:02:44 am

DJ Bazzie Wazzie
Member
From: the Netherlands
Registered: 2004-10-20
Posts: 2648

Re: Making Filename Searches Lightening Fast

pcLoadLetter wrote:

How do I pass the information from Javascript back to AppleScript? The search is working from Bridge. I just need to make the results accessible from AppleScript

What does the dictionary say? Mine says that the do javascript command can only return an integer, meaning you can't data from the JavaScript runtime into AppleScript so you have to write it to an external file and read it with AS from there.

I'm not aware if you noticed but the JavaScript getFiles() function isn't recursive as you preferred.

Offline

 

#15 2017-05-03 08:34:54 am

t.spoon
Member
From: BFE, Massachusetts
Registered: 2013-01-12
Posts: 185

Re: Making Filename Searches Lightening Fast

Not really, I don't know Javascript except for very minimally. I've just seen it in places around the web that Javascript and Applescript were similar, and that they were both considered to be highly influenced by Smalltalk. At work they've twice decided to have another developer learn some Applescript, and the development department chose people who already knew Javascript because they considered them similar. But I don't know enough about it to get into specifics.

In this particular context the reason I was mentioning it is that it was my understanding that, like Applescript, Javascript maintains some piece of information that's "the last result" that is the current output. The way Script Editor shows you a "result" at the bottom, and that does not only show something based on a "return" statement at the end of the program. "Set bob to 2" will return 2. Handlers don't need an explicit "return" line at the end, if the last thing set or calculated was what you want to return anyway.

So in this instance where you want to return a value from Javascript to Applescript, it was my understanding that the "returned" value from a Javascript handler is very similar to whatever would show up as the "returned" value for an Applescript doing the same thing.

But experimentally, it looks like I was wrong about that anyway.

Applescript:

set theOutput to myFunction()

on myFunction()
   set x to 2
end myFunction

Sets theOutput to 2.

or

Applescript:

set theOutput to run script "set x to 2"

also sets theOutput to 2.

I'll use Photoshop to run my Javascripts, because it looks like Applescript can't run Javascript inline unless it's inside a "tell" to an Application that runs Javascript. It looks like otherwise it has to be a separate file.

Applescript:

tell application "Adobe Photoshop CC 2017"
   set theOutput to do javascript "
var x = 2;
"

end tell

theOutput remains undefined.

Applescript:

tell application "Adobe Photoshop CC 2017"
   set theOutput to do javascript "
var x = 2;
return x
"

end tell

Causes an error because a "return" has to be inside a function in Javascript.

Applescript:

tell application "Adobe Photoshop CC 2017"
   set theOutput to do javascript "
myFunction()
function myFunction() {
var x = 2;
}
"

end tell

theOutput is once again undefined.

Applescript:

tell application "Adobe Photoshop CC 2017"
   set theOutput to do javascript "
myFunction()
function myFunction() {
var x = 2;
return x
}
"

end tell

Sets theOutput to 2. So nevermind my previous post, it looks like the values you want to return to Applescript need to be explicitly returned at the end of a Javascript function.

So (WARNING - I don't know Javascript and don't have Bridge to test this) - it would appear to me that your script should go something like this:

Applescript:

set high_res_folder to "/Volumes/sblain$/*SIGNS FOR PREPRESS/Agriculture Dept/Chicken Coops/high res images/"

tell application "Adobe Bridge CC 2017"
   activate
   
   set myFiles to do javascript "
bridgeSearch()

function bridgeSearch() {
   #target bridge;
   
   var search_folder = ['"
& high_res_folder & "'];
   alert(search_criteria);
var my_files = Folder(search_folder).getFiles('*811951*');
return my_files;
}
"

   
end tell


Hackintosh built February, 2012 |  Mac OS Sierra
GIGABYTE GA-Z68X-UD3H-B3 | Core i5 2500k | 16 GB DDR3 | GIGABYTE Geforce 1050 TI 4GB
250 GB Samsung 850 EVO | 4 TB RAID
Dell Ultrasharp U3011 | Dell Ultrasharp 2007FPb

Offline

 

#16 2017-05-03 09:52:56 am

DJ Bazzie Wazzie
Member
From: the Netherlands
Registered: 2004-10-20
Posts: 2648

Re: Making Filename Searches Lightening Fast

t.spoon wrote:

So (WARNING - I don't know Javascript and don't have Bridge to test this) - it would appear to me that your script should go something like this:

I have bridge installed and the dictionary says it can only return an integer, state of script. I was stubborn and tried it my self because wrapping it in an function and return it's value works in the ExtendScript Toolkit however it doesn't work in do javascript (at least not in my version).

Offline

 

#17 2017-05-03 11:19:51 am

t.spoon
Member
From: BFE, Massachusetts
Registered: 2013-01-12
Posts: 185

Re: Making Filename Searches Lightening Fast

Well, that sounds like the end of this approach if the OP needs the search results back in Applescript.

Weird it only allows returning integers. "do javascript" in Photoshop allows returning different things, certainly strings.

If he's OK with the search results simply being displayed inside a Bridge window, but AS not knowing about them, that's probably doable. If AS needs the results, then I think it's time to move onto another option, probably spotlight with Applescript Toolbox to constrain the search folder but allow recursion inside it.

- Tom.


Hackintosh built February, 2012 |  Mac OS Sierra
GIGABYTE GA-Z68X-UD3H-B3 | Core i5 2500k | 16 GB DDR3 | GIGABYTE Geforce 1050 TI 4GB
250 GB Samsung 850 EVO | 4 TB RAID
Dell Ultrasharp U3011 | Dell Ultrasharp 2007FPb

Offline

 

#18 2017-05-03 12:12:24 pm

DJ Bazzie Wazzie
Member
From: the Netherlands
Registered: 2004-10-20
Posts: 2648

Re: Making Filename Searches Lightening Fast

t.spoon wrote:

I've just seen it in places around the web that Javascript and Applescript were similar, and that they were both considered to be highly influenced by Smalltalk. At work they've twice decided to have another developer learn some Applescript, and the development department chose people who already knew Javascript because they considered them similar. But I don't know enough about it to get into specifics.

There are some similarities between AppleScript and JavaScript because the parent property and prototype model is adopted in JavaScript later, or correctly said in ECMAScript. The decision they made at work is therefore understandable and I think JavaScript is one of the closest languages to AppleScript when it comes to its object model.  The influences of Hypertalk and Self created these similarities but these similarities doesn't come from Smalltalk


Applescript:

tell application "Adobe Photoshop CS6"
   activate
   
   do javascript "
function findFiles(theFolder){
var theFiles = theFolder.getFiles (\"*i*\");
return theFiles
}

var theFolder = Folder (\"/usr\");
findFiles(theFolder);"

end tell

This does work in PhotoShop and return an comma separated list

Last edited by DJ Bazzie Wazzie (2017-05-03 12:15:37 pm)

Offline

 

#19 2017-05-03 04:06:00 pm

t.spoon
Member
From: BFE, Massachusetts
Registered: 2013-01-12
Posts: 185

Re: Making Filename Searches Lightening Fast

it can only return an integer

The javascript function could encode the search results into a single (very large) integer and then Applescript could decode them back out. tongue


Hackintosh built February, 2012 |  Mac OS Sierra
GIGABYTE GA-Z68X-UD3H-B3 | Core i5 2500k | 16 GB DDR3 | GIGABYTE Geforce 1050 TI 4GB
250 GB Samsung 850 EVO | 4 TB RAID
Dell Ultrasharp U3011 | Dell Ultrasharp 2007FPb

Offline

 

Board footer

Powered by FluxBB

[ Generated in 0.054 seconds, 9 queries executed ]

RSS (new topics) RSS (active topics)