Friday, April 28, 2017

#1 2017-04-20 02:56:40 pm

pcLoadLetter
Member
Registered: 2016-09-23
Posts: 12

Making Filename Searches Lightening Fast

This script searches through several different directories (and their subdirectories). The problem is that I'm searching through so many different files in the last 2 directories, which makes the shellscript far slower. (I went from the search taking seconds per image to it taking up to 5 to 10 minutes for an image that isn't found in the smaller directories. to several hours when I added those two directories to the script. This script loops through hundreds or even a thousand plus files, so I need to make it faster to work with our workflow).

What can I do to make automating searching through large directories faster? I'm open to using a different shell script command or language. Any advice is greatly appreciated. Thanks!


Applescript:

--Stringing together the missing image to search for
set search_name to "*" & blain_number & "*" as text


--Specifying the path to the user's hard drive
set my_HD to path to startup disk as string

--Searching for the images
tell application "Finder"
set image_search1 to paragraphs of (do shell script "find " & quoted form of (POSIX path of images_folder) & " -name '" & search_name & "' -type f") -->searching
if (image_search1 as string) is "" then
set image_search1 to paragraphs of (do shell script "find " & quoted form of (POSIX path of processing_folder) & " -name '" & search_name & "' -type f") -->searching
end if
if (image_search1 as string) is "" then
set image_search1 to paragraphs of (do shell script "find " & quoted form of (POSIX path of r_drive_folder) & " -name '" & search_name & "' -type f") -->searching
end if
if (image_search1 as string) is "" then
set image_search1 to paragraphs of (do shell script "find " & quoted form of (POSIX path of adv_print_folder) & " -name '" & search_name & "' -type f") -->searching
end if
end tell


--Adding the image to the found link list
if (image_search1 as string) is not "" then
set found_image_list to found_image_list & image_search1
set found_number_list to found_number_list & blain_number
else
set missing_link_list to missing_link_list & blain_number
end if

Offline

 

#2 2017-04-20 04:57:24 pm

Marc Anthony
Member
From: Dallas, TX
Registered: 2006-04-27
Posts: 740

Re: Making Filename Searches Lightening Fast

Hi. There is little you can do to speed up the shell script except limit the traversal's depth, as searching through jillions of files and folders recursively takes time. You do need to nix the Finder call; it has nothing to do with the shell.

Offline

 

#3 2017-04-20 06:28:50 pm

pcLoadLetter
Member
Registered: 2016-09-23
Posts: 12

Re: Making Filename Searches Lightening Fast

What about trying something other than shellscript? Bridge seems to find the files by hand faster and it likes javascript. Maybe there's something in JavaScript or another language? I think searching through less files would be a great idea! The hard part is convincing other people to implement a better organized image library. The code is starting to seem easy by comparison. lol

Last edited by pcLoadLetter (2017-04-20 06:45:06 pm)

Offline

 

#4 2017-04-20 06:53:40 pm

Shane Stanley
Member
From: Australia
Registered: 2002-12-07
Posts: 4998

Re: Making Filename Searches Lightening Fast

What Marc says. And if you're searching on remote volumes, which it sounds like, doubly so. You can speed up searches of remote volumes by caching the contents locally, but that can work only when the contents are not changing too often, and it also complicates things.


Shane Stanley <sstanley@myriad-com.com.au>
www.macosxautomation.com/applescript/apps/

Offline

 

#5 2017-04-24 04:37:58 pm

t.spoon
Member
From: BFE, Massachusetts
Registered: 2013-01-12
Posts: 130

Re: Making Filename Searches Lightening Fast

Spotlight may offer a speed improvement over shell "find." There is some talk of remote volumes, in which case they might not be Spotlight indexed. But if they're local, or remote volumes hosted on a Mac and shared via AFP supporting remote Spotlight queries, then I'd expect using Spotlight instead of UNIX "Find" to be a lot faster. Examples of Applescript & Spotlight in this thread:

http://macscripter.net/viewtopic.php?id=27833

Another possible alternative for a remote volume shared by AFP that's not Spotlight Searchable is to script Find Any File. http://apps.tempel.org/FindAnyFile/ There's no Applescript dictionary, but it looks pretty amenable to UI scripting.

But if it's a remote SMB volume, then neither of these will be of any help speed-wise.

You mentioned Bridge is faster - Bridge builds its own index like spotlight. If Bridge works in your scenario, then yes, it is scriptable with Javascript. And if you want to stick with everything but the search being in Applescript, Bridge has one Applescript command: "do javascript."

Another option, if it is a remote volume shared by something other than AFP and you are OK with allowing the overhead of a local index of a remote volume, is to use Quicksilver to index the required parts of the remote volume, then Applescript a Quicksilver search. https://qsapp.com/ That's as fast as spotlight when searching and works with remote SMB/NFS/webdav volumes. Quicksilver has a good Applescript Dictionary.

If this is confusing, let me know how this volume is accessed (local or remote and, if remote, the protocol), and whether or not you have Bridge indexing it anyway, and I can help you zoom in on possibly strategies.

Offline

 

#6 2017-04-25 10:47:52 am

pcLoadLetter
Member
Registered: 2016-09-23
Posts: 12

Re: Making Filename Searches Lightening Fast

Thanks! I like the Bridge idea. I was having trouble getting the javascript right for the search. Here's what I found in the javascript guide and javascript reference: Any help would be greatly appreciated. (I had it search my desktop to test it, but it will end up searching a server connected by smb).

Applescript:

set my_folder to "/Users/user_name/Desktop/"
set my_search_criteria to "612369"

tell application "Adobe Bridge CS5.1"
   activate
   
   do javascript "
               #target 'bridge'
//Declaring the variables for the folder and search criteria
var search_folder = ['' & my_folder & ''];
var search_criteria = ['' & my_search_criteria & ''];

new SearchCriteria (searchField, operandType,
*searchFieldDisplay, *operands);

new SearchDefinition (criteriaList, defaultResultsLimit,
*ranks, *scopeSpecifiers);

new SearchSpecification (conditionList, conjunction, maximumResults,
rankOrdering, rankField, scopeSpecifiers);

obj.getBridgeURIForSearch
(scope, specification)

obj.getSearchDetails ()

/**/"

end tell

Last edited by pcLoadLetter (2017-04-25 10:49:45 am)

Offline

 

#7 2017-04-25 12:39:40 pm

t.spoon
Member
From: BFE, Massachusetts
Registered: 2013-01-12
Posts: 130

Re: Making Filename Searches Lightening Fast

It may be time to move to a javascript forum. Javascripting Bridge may be the best approach in your scenario, but you're on an Applescript forum... you may want to try the Adobe Bridge scripting forums:
https://forums.adobe.com/community/brid … _scripting
or Stack Overflow:
http://stackoverflow.com/

Maybe someone here also Javascripts and will chime in with help, but I wouldn't expect it on an Applescript forum.

Best of luck with it though,

Tom.

Offline

 

#8 2017-04-25 06:26:16 pm

JMichaelTX
Member
From: Houston, TX (The Woodlands)
Registered: 2014-07-12
Posts: 131

Re: Making Filename Searches Lightening Fast

pcLoadLetter wrote:

This script searches through several different directories (and their subdirectories). The problem is that I'm searching through so many different files in the last 2 directories, which makes the shellscript far slower. (I went from the search taking seconds per image to it taking up to 5 to 10 minutes for an image that isn't found in the smaller directories. to several hours when I added those two directories to the script. This script loops through hundreds or even a thousand plus files, so I need to make it faster to work with our workflow).

What can I do to make automating searching through large directories faster? I'm open to using a different shell script command or language. Any advice is greatly appreciated. Thanks!

I'm curious.  Have you tried a manual Spotlight search, just using the part of the file name you are interested in?
If you only want image files, then you could limit the search by using "kind:image" (without the quotes).
Spotlight finds files near instantaneously for me on an iMac-27 with 2TB Fusion drive AND a 3TB LaCie ThunderBolt2 drive.

If Spotlight is quick enough for you, you can use the `mdfind` command in a shell script.
See:
mdfind Man Page - macOS - SS64.com
Spotlight syntax, mdfind examples, and metadata attributes

There are also some ASObjC searches that are very fast, but @ShaneStanley is better equipped to talk about those than I am.


iMac-27 Late 2015 Retina 5K Screen (& others)
macOS 10.11.6 (El Capitan)

Offline

 

Board footer

Powered by FluxBB

[ Generated in 0.142 seconds, 8 queries executed ]

RSS (new topics) RSS (active topics)