Simple Script...Long List Table Overflow?

mcsprodart · January 11, 2024, 1:25pm

I haven’t experienced a table overflow on such a simple script. It seems to be if I have two simple lists of 3000-4000 strings. Any advice? Ironically when I tried to create this topic it errors with An error occurred: Body is limited to 32000 characters; you entered 76616. lol. So you’ll have to increase the size of these lists to 3000-4000 items to test.

use AppleScript version "2.4" -- Yosemite (10.10) or later
use scripting additions

set wtf to "6405CVC"
set theResult to my testVendors(wtf)

on testVendors(testStyle)

set vendorAPI to missing value

---list one here needs more items
set allSSStyles to {"4700", "4710", "N3142", "N3165", "N3373", "N3381", "N3409", "N3425", "N4279", "N5293", "N5296", "N5338", "NB3142", "NB5301", "NW3002", "NW3201", "NW3402", "EB101", "LP101", "A1000", "A12S", "A130", "A131", "A166", "A186", "A190", "A191", "A207", "A213", "A230", "A231", "A233", "A236", "A240", "A241", "A267", "A268", "A280", "A281", "A284", "A285", "A295", "A322", "A323", "A324", "A325", "A373", "A376", "A377", "A401", "A402", "A403", "A416", "A417", "A430", "A431", "A432"}
--list 2 here needs more items
set allSanmarStyles to {"29M", "4662M", "5170", "5180", "5186", "5190", "5280", "5370", "562M", "P160", "P170", "B100", "4850MP", "4997M", "B300", "B050", "562B", "4528M", "S100", "BTU", "973M", "F260", "S600", "S600T", "996Y", "054X", "29B", "PWU", "S500T", "SP10", "993M", "437M", "HCF", "B400", "K420", "SP11", "K320", "436MP", "J730", "PC61", "5250", "L420", "K431", "J753", "J754", "TLJ754", "PC61Y", "PC61P", "PC61LS", "996M", "C800", "J777", "K420P", "TLS600", "TLS600T", "PC90", "PC90H", "PC55"}

if allSanmarStyles contains testStyle then
	set vendorAPI to "Sanmar"
else if allSSStyles contains testStyle then
	set vendorAPI to "SSActivewear"
end if
return vendorAPI

end testVendors

`

peavine · January 11, 2024, 2:20pm

The following works on my Sonoma computer with lists that exceed 3000 items. I noticed that you use the my reserved word in calling the handler–perhaps the error arises elsewhere in the script. BTW, I moved the lists out of the handler for convenience to make them larger.

use AppleScript version "2.4" -- Yosemite (10.10) or later
use scripting additions

set testStyle to "6405CVC"

set allSSStyles to {"4700", "4710", "N3142", "N3165", "N3373", "N3381", "N3409", "N3425", "N4279", "N5293", "N5296", "N5338", "NB3142", "NB5301", "NW3002", "NW3201", "NW3402", "EB101", "LP101", "A1000", "A12S", "A130", "A131", "A166", "A186", "A190", "A191", "A207", "A213", "A230", "A231", "A233", "A236", "A240", "A241", "A267", "A268", "A280", "A281", "A284", "A285", "A295", "A322", "A323", "A324", "A325", "A373", "A376", "A377", "A401", "A402", "A403", "A416", "A417", "A430", "A431", "A432"}

set allSanmarStyles to {"29M", "4662M", "5170", "5180", "5186", "5190", "5280", "5370", "562M", "P160", "P170", "B100", "4850MP", "4997M", "B300", "B050", "562B", "4528M", "S100", "BTU", "973M", "F260", "S600", "S600T", "996Y", "054X", "29B", "PWU", "S500T", "SP10", "993M", "437M", "HCF", "B400", "K420", "SP11", "K320", "436MP", "J730", "PC61", "5250", "L420", "K431", "J753", "J754", "TLJ754", "PC61Y", "PC61P", "PC61LS", "996M", "C800", "J777", "K420P", "TLS600", "TLS600T", "PC90", "PC90H", "PC55"}

-- make large lists
repeat 6 times
	set allSSStyles to allSSStyles & allSSStyles
	set allSanmarStyles to allSanmarStyles & allSanmarStyles
	# count allSSStyles --> 3648
	# count allSanmarStyles --> 3712
end repeat

-- create match for testing
set end of allSanmarStyles to "6405CVC"

set theResult to testVendors(testStyle, allSSStyles, allSanmarStyles) --> "Sanmar"

on testVendors(testStyle, allSSStyles, allSanmarStyles)
	set vendorAPI to missing value
	if allSanmarStyles contains testStyle then
		set vendorAPI to "Sanmar"
	else if allSSStyles contains testStyle then
		set vendorAPI to "SSActivewear"
	end if
	return vendorAPI
end testVendors

leo_r · January 11, 2024, 4:29pm

Try to store you lists in two text files then read the files. I think it would be a more appropriate approach for such long lists anyway (rather than hardcoding thousands of items in your script).

robertfern · January 11, 2024, 5:07pm

Odd, I increased the size of your lists to over 6000 each and It worked fine, no issue.

use AppleScript version "2.4" -- Yosemite (10.10) or later
use scripting additions

set wtf to "6405CVC"
set theResult to my testVendors(wtf)

on testVendors(testStyle)
	local allSSStyles, allSanmarStyles
	set vendorAPI to "none"
	
	---list one here needs more items
	set allSSStyles to {"4700", "4710", "N3142", "N3165", "N3373", "N3381", "N3409", "N3425", "N4279", "N5293", "N5296", "N5338", "NB3142", "NB5301", "NW3002", "NW3201", "NW3402", "EB101", "LP101", "A1000", "A12S", "A130", "A131", "A166", "A186", "A190", "A191", "A207", "A213", "A230", "A231", "A233", "A236", "A240", "A241", "A267", "A268", "A280", "A281", "A284", "A285", "A295", "A322", "A323", "A324", "A325", "A373", "A376", "A377", "A401", "A402", "A403", "A416", "A417", "A430", "A431", "A432"}
	--list 2 here needs more items
	set allSanmarStyles to {"29M", "4662M", "5170", "5180", "5186", "5190", "5280", "5370", "562M", "P160", "P170", "B100", "4850MP", "4997M", "B300", "B050", "562B", "4528M", "S100", "BTU", "973M", "F260", "S600", "S600T", "996Y", "054X", "29B", "PWU", "S500T", "SP10", "993M", "437M", "HCF", "B400", "K420", "SP11", "K320", "436MP", "J730", "PC61", "5250", "L420", "K431", "J753", "J754", "TLJ754", "PC61Y", "PC61P", "PC61LS", "996M", "C800", "J777", "K420P", "TLS600", "TLS600T", "PC90", "PC90H", "PC55"}
	
	repeat 7 times
		set allSSStyles to allSSStyles & allSSStyles
		set allSanmarStyles to allSanmarStyles & allSanmarStyles
	end repeat
	set end of allSanmarStyles to "6405CVC" -- to create match
	
	if allSanmarStyles contains testStyle then
		set vendorAPI to "Sanmar"
	else if allSSStyles contains testStyle then
		set vendorAPI to "SSActivewear"
	end if
	return vendorAPI
end testVendors

I used local lists that were explicitly defined. Dont’ know if that had an effect.

mcsprodart · January 11, 2024, 6:30pm

@robertfern It compiles when I add local lists and wouldn’t before. I didn’t think you had to declare local in subroutine anymore. I thought all were local unless declared properties in main portion of script. It does not however run or save. I think the difference is yours has 57 items in the list initially. I think the only way I can resolve is by reading a Plist or Text file or taking the time to check the APIs if the item is offered first since that was my initial goal. It just takes a few seconds each API and thought it would be faster to have a list of items for each vendor. So much for faster…

mcsprodart · January 11, 2024, 6:34pm

@leo_r I believe you’re right about this being the best option unless I store in Plist but either way reading file and looping through 7000 items in list is probably going to take just as long as checking the API for product which was trying to eliminate the few seconds it takes on each. I’ll try and speed up the API check I am doing for now. Thank you for the suggestion.

robertfern · January 11, 2024, 8:08pm

Can you post or send me a text file of the lists you are trying to use?

I frequently use large lists saved in raw AppleScript format to text files, which I then import with one line.

peavine · January 12, 2024, 2:18am

The OP has decided to take a different approach, but I wondered how fast this could be done with ASObjC. With over 14,000 items in each of the two text files, and with the matching string at the end of the second text file, this script took 8 milliseconds to run. This is with the Foundation framework in memory, which would normally be the case.

use framework "Foundation"
use scripting additions

set searchStyle to "6405CVC"

set theStyle to getStyle(searchStyle)

on getStyle(searchStyle)
	set fileOne to POSIX path of (path to desktop) & "allSSStyles.txt" -- user set to desired value
	set fileTwo to POSIX path of (path to desktop) & "allSanmarStyles.txt" -- user set to desired value
	set stringOne to current application's NSString's stringWithContentsOfFile:fileOne encoding:(current application's NSUTF8StringEncoding) |error|:(missing value)
	set arrayOne to (stringOne's componentsSeparatedByString:linefeed)
	if (arrayOne's containsObject:searchStyle) is true then return "SSActivewear"
	set stringTwo to current application's NSString's stringWithContentsOfFile:fileTwo encoding:(current application's NSUTF8StringEncoding) |error|:(missing value)
	set arrayTwo to (stringTwo's componentsSeparatedByString:linefeed)
	if (arrayTwo's containsObject:searchStyle) is true then return "Sanmar"
	return "No Match"
end getStyle

robertfern · January 12, 2024, 2:42am

I would like to get it running natively, but I’m waiting on mcsprodart to post the actual string lists he’s woking with.

peavine · January 12, 2024, 2:58pm

FWIW, the following is the script I wrote to create the test files. It uses the lists contained in post 1 and simply duplicates them for testing purposes.

use framework "Foundation"
use scripting additions

set fileOne to POSIX path of (path to desktop) & "allSSStyles.txt"
set fileTwo to POSIX path of (path to desktop) & "allSanmarStyles.txt"

set listOne to {"4700", "4710", "N3142", "N3165", "N3373", "N3381", "N3409", "N3425", "N4279", "N5293", "N5296", "N5338", "NB3142", "NB5301", "NW3002", "NW3201", "NW3402", "EB101", "LP101", "A1000", "A12S", "A130", "A131", "A166", "A186", "A190", "A191", "A207", "A213", "A230", "A231", "A233", "A236", "A240", "A241", "A267", "A268", "A280", "A281", "A284", "A285", "A295", "A322", "A323", "A324", "A325", "A373", "A376", "A377", "A401", "A402", "A403", "A416", "A417", "A430", "A431", "A432"}

set listTwo to {"29M", "4662M", "5170", "5180", "5186", "5190", "5280", "5370", "562M", "P160", "P170", "B100", "4850MP", "4997M", "B300", "B050", "562B", "4528M", "S100", "BTU", "973M", "F260", "S600", "S600T", "996Y", "054X", "29B", "PWU", "S500T", "SP10", "993M", "437M", "HCF", "B400", "K420", "SP11", "K320", "436MP", "J730", "PC61", "5250", "L420", "K431", "J753", "J754", "TLJ754", "PC61Y", "PC61P", "PC61LS", "996M", "C800", "J777", "K420P", "TLS600", "TLS600T", "PC90", "PC90H", "PC55"}

repeat 8 times
	set listOne to listOne & listOne
	set listTwo to listTwo & listTwo
	# count listOne --> 14592
	# count listTwo --> 14848
end repeat

set end of listTwo to "6405CVC" -- to create match

set arrayOne to current application's NSArray's arrayWithArray:listOne
set arrayTwo to current application's NSArray's arrayWithArray:listTwo
set stringOne to arrayOne's componentsJoinedByString:linefeed
set stringTwo to arrayTwo's componentsJoinedByString:linefeed
stringOne's writeToFile:fileOne atomically:true encoding:(current application's NSUTF8StringEncoding) |error|:(missing value)
stringTwo's writeToFile:fileTwo atomically:true encoding:(current application's NSUTF8StringEncoding) |error|:(missing value)

robertfern · January 12, 2024, 4:17pm

That’s what I did. And it ran fine and fast.

I’m trying to decipher what mcsprodart is doing differently?

peavine · January 12, 2024, 4:46pm

Agreed–I never understood why mcsprodart was getting an error. My guess was that there was some issue elsewhere in his script. He’s decided to use a different approach, so I suspect this will remain a mystery.

robertfern · January 12, 2024, 7:31pm

Peavine,

I combined your two ApplescriptObjC scripts and removed the exporting to text files, to keep everything in memory

use framework "Foundation"
use scripting additions

set listOne to {"4700", "4710", "N3142", "N3165", "N3373", "N3381", "N3409", "N3425", "N4279", "N5293", "N5296", "N5338", "NB3142", "NB5301", "NW3002", "NW3201", "NW3402", "EB101", "LP101", "A1000", "A12S", "A130", "A131", "A166", "A186", "A190", "A191", "A207", "A213", "A230", "A231", "A233", "A236", "A240", "A241", "A267", "A268", "A280", "A281", "A284", "A285", "A295", "A322", "A323", "A324", "A325", "A373", "A376", "A377", "A401", "A402", "A403", "A416", "A417", "A430", "A431", "A432"}

set listTwo to {"29M", "4662M", "5170", "5180", "5186", "5190", "5280", "5370", "562M", "P160", "P170", "B100", "4850MP", "4997M", "B300", "B050", "562B", "4528M", "S100", "BTU", "973M", "F260", "S600", "S600T", "996Y", "054X", "29B", "PWU", "S500T", "SP10", "993M", "437M", "HCF", "B400", "K420", "SP11", "K320", "436MP", "J730", "PC61", "5250", "L420", "K431", "J753", "J754", "TLJ754", "PC61Y", "PC61P", "PC61LS", "996M", "C800", "J777", "K420P", "TLS600", "TLS600T", "PC90", "PC90H", "PC55"}

repeat 8 times
	set listOne to listOne & listOne
	set listTwo to listTwo & listTwo
end repeat

set end of listTwo to "6405CVC" -- to create match
set searchStyle to "6405CVC"

set arrayOne to current application's NSArray's arrayWithArray:listOne
set arrayTwo to current application's NSArray's arrayWithArray:listTwo
if (arrayOne's containsObject:searchStyle) is true then return "SSActivewear"
if (arrayTwo's containsObject:searchStyle) is true then return "Sanmar"
return "No Match"

ApplescriptObjC version was 4.02 times as slow as AppleScript only version

In a lot of cases, ApplescriptObjC is not as fast as everyone thinks it is.
There is a lot of overhead in loading the frameworks, then there is more overhead in converting large AppleScript lists to arrays in ObjC and back again.

mcsprodart · January 12, 2024, 11:00pm

@peavine @Fredrik71 I still dont get it either. Here’s a text file of the actual script and a text file of each list for reference. This should be simple. I totally get reading a file might be more efficient especially if I dont loop through AppleScript to find a match using filteredArrayUsingPredicate it should work either way and doesn’t. When compiling it takes forever but can’t be saved or run in Script Debugger or Script Editor. Ive restarted it definitely seems to be a limit I didn’t realize existed.

The whole intent of this was to be faster than actually searching via the Vendor APIs in ApplescriptObjc waiting for missing values to be returned before trying another. Amazingly enough I couldn’t even look in a list of 8000 items for a match in AppleScript. Maybe one of you can shed some light on it. As for the solution for now I fell back on searching the multiple APIs until I find a match then use the API to grab info using applescriptObjc.

I edited my post and attached the text files to use with @peavine script and the standard AppleScript as a text file that will not run or save.

TestingFiles.zip (55.6 KB)

mcsprodart · January 12, 2024, 11:15pm

@peavine Testing your script now its insanely fast and works as expected. Thank you for helping me realize how much faster reading a file is. Usually I read a tab delineated text file of multiple columns and generally it’s not very fast but never long enough to care. I suspected would be “faster” reading text file however didn’t expect the standard search list of strings in AppleScript to not work at all. tt still baffles me why a list of 8000 strings would blow things up. I plan on updating current solution to check the text files instead of APIs and it will definately speed things up. For the record your script takes 1/10 of s second in Script Debugger…holy moly.

robertfern · January 13, 2024, 12:42am

Fredrik71,

I’m not sure what you are trying to tell me?

peavine · January 13, 2024, 12:47am

mcsprodart. I tested your script in Script Debugger and received the same error. So, the issue is not the number of items in the lists but the number of items in the AppleScript

The best option appears to be the use of a text file. A second option–which worked in my testing but remains problematic–is to use strings in the script, which can then be made into lists with text item delimiters. I tested the following with the full lists and it worked, but saving the script was slow:

use AppleScript version "2.4" -- Yosemite (10.10) or later
use scripting additions

set testStyle to "6405CVC"
set theResult to my testVendors(testStyle)

on testVendors(testStyle)
	set TID to text item delimiters
	set text item delimiters to space
	
	set allSanmarStyles to "29M 4662M 5170 5180" -- abbreviated for posting
	set allSSStyles to "4700 4710 N3142 N3165 6405CVC" -- abbreviated for posting
	
	set allSanmarStyles to text items of allSanmarStyles
	set allSSStyles to text items of allSSStyles
	
	if allSanmarStyles contains testStyle then
		set vendorAPI to "Sanmar"
	else if allSSStyles contains testStyle then
		set vendorAPI to "SSActivewear"
	end if
	set text item delimiters to TID
	return vendorAPI
end testVendors

mcsprodart · January 13, 2024, 1:01am

Nice. Still confuses me why a list of 8000 items could cause a table overflow but converting it to a list using text items could make a difference. I have much much longer scripts that push the limits of table overflow enough to have to remove commented out code to save. Both your solutions are great and make things easier to setup. I’m sticking with reading the files since they could change over time anyhow. Thanks again!

peavine · January 13, 2024, 1:16am

I share your lack of understanding on this issue. However, I think the answer may be in the following, which was written by Mark (here):

One of the most infuriating AppleScript errors is also one that many users see at some stage: errOSAInternalTableOverflow. It is particularly annoying because when it happens AppleScript itself is usually corrupted, so you have to close and relaunch your editor…

Simply, the error means a script has too many objects to keep track of. When a script is stored, AppleScript assigns every object an ID, which is used to reconstruct references between objects when they are loaded back in. There is a maximum number of IDs available for any given store/load operation. If a script has too many objects, the ID table overflows and you get this error. [emphasis mine]

peavine · January 13, 2024, 1:33am

@robertfern. I included in post 10 a script that I thought might be of use to the OP or to anyone else with a similar need. That script took 8 milliseconds to run. If you want to criticize one of my scripts, criticize that one.

I posted a separate script in post 12, and the only purpose of that script was to create text files for testing purposes. In writing this script, I gave no consideration to how long it would take to run.

You took part of my first script and combined it with part of my second script with the obvious intent to criticize ASObjC. I understand that you dislike ASObjC, but this is becoming a bit much.