Internal table overflow. (error -2707)

Hi,

Has anyone come across this error, Internal table overflow. (error -2707), and what does it mean?

I get it in a repeat loop that may be called 100’s of times summing values extracted from NSMutableDictionary then written back on another key.

I thought maybe that garbage collection was not keeping up with the creation of dictionaries?

Is it necessary to clear the dictionaries created at the end of each repeat?

Any ideas please.

Also is there anyway to speed up the process of repeating through the loop to carryout the calculation on 100’s of table rows before reloading the data into the table.

I did try implementing a single row calculation in the tables delegate routine tableView_objectValueForTableColumn_row_(aTableView, aTableColumn, rowIndex)

but it did slow the scrolling of the field.

Any ideas on this also gratefully received.

	set tCount to tRecordData's |count|()
		--log "number of records = " & tCount
		repeat with tRowIndex from 0 to tCount - 1
			--log "Record Number = " & tRowIndex
			set tThisRecDict to tRecordData's objectAtIndex_(tRowIndex)
			set tThisRecValue to tThisRecDict's valueForKey_("amountA") as real
			if tRowIndex = 0 then
				tThisRecDict's setValue_forKey_(tThisRecValue, "amountB")
			else
				set tPrevRecDict to tRecordData's objectAtIndex_(tRowIndex - 1)
				set tPrevRecBalance to tPrevRecDict's valueForKey_("amountB") as real
				set tThisRecBalance to (tPrevRecBalance + tThisRecValue) as real
				tThisRecDict's setValue_forKey_(tThisRecBalance, "amountB")
			end if
		end repeat

I doubt it’s anything to do with garbage collection. How are you triggering it to happen? I wonder if what is happening is that something else is being triggered before the repeat loop is complete.

Off the top of my head, I can’t think of any obvious way to speed it up short of moving it to Objective-C. Perhaps keeping the data as an array of arrays, and having your datasource handlers do the conversion to dictionaries, would help – but there’s no way of knowing without trying it.

Actually, this might speed things up a tad:

set tCount to tRecordData's |count|()
--log "number of records = " & tCount
set tThisRecDict to tRecordData's objectAtIndex_(0)
set tPrevRecBalance to tThisRecDict's valueForKey_("amountA") as real
tThisRecDict's setValue_forKey_(tPrevRecBalance, "amountB")
repeat with tRowIndex from 1 to tCount - 1
	set tThisRecDict to tRecordData's objectAtIndex_(tRowIndex)
	set tThisRecValue to tThisRecDict's valueForKey_("amountA") as real
	set tPrevRecBalance to (tPrevRecBalance + tThisRecValue)
	tThisRecDict's setValue_forKey_(tPrevRecBalance, "amountB")
end repeat

I don’t know if using the “repeat with anEntry in theData” type of enumeration is faster or not. I tried the following program, which creates 10,000 records with random numbers in them. The makeTotals_ method is connected to a button, and when I click it, it takes about 4 seconds to create the running totals in the second column – that seems pretty fast to me (I’m using bindings to populate my table).

script TransactionsAppDelegate
	property parent : class "NSObject"
	property theData : missing value
	
	on applicationWillFinishLaunching_(aNotification)
		set theData to current application's NSMutableArray's alloc()'s init()
		repeat 10000 times
			set dAmount to (random number from 100 to 10000) / 100 as real
			theData's addObject_({amountA:dAmount})
		end repeat
		setTheData_(theData)
	end applicationWillFinishLaunching_
	
	on makeTotals_(sender)
		set theTotal to 0
		repeat with anEntry in theData
			set theTotal to theTotal + (anEntry's valueForKey_("amountA") as real)
			anEntry's addObject_forKey_(theTotal, "amountB")
		end repeat
	end makeTotals_
	
end script

Ric

Hi,

Thanks for the responses as it is Christmas Day here my better half has put the block on so I will not be able to try your suggestions for a day or so.

I will report back.

Shane’s suggestion will shave some time off by omitting the “if” check through the loop.

Actually the data is also stored in a SQLite table. As I am just learning SQLite I was wondering if there was any mileage there?

Al the best

Terry

Just to see how much difference it would make, I tested an Objective C version of my program, so I replaced the code in makeTotals_ with this Objective C class, and made my AS a subclass of it:


#import "ArrayEnum.h"

@implementation ArrayEnum

-(void)makeTotals:(NSMutableArray *)myArray{
float total=0;
for (id obj in myArray){
    total=total + [[obj valueForKey:@"amountA"] floatValue];
    [obj setValue:[NSNumber numberWithFloat:total] forKey:@"amountB"];
}
}   

@end

I tested this with 100,000 rows of data – the AS way took 48 seconds to make the totals column, the Objective C method took about 1/2 a second.

Ric

That’s quite a difference…

Hi,

Taking your hint I produced this routine applicable to what I am doing and it sums over 750 records instantaneously:

Objective-C seems to be the answer:


+(void)makeTotals:(NSMutableArray *)myArray
{
	int n;
	float tThisRecBalance;
	for (n=0;n < [myArray count];n++)
	{
		if (n==0)
		{
			[[myArray objectAtIndex:n] setValue:[[myArray objectAtIndex:n] valueForKey:@"amount"] forKey: @"balance"];
		}
		else
		{
			tThisRecBalance = [[[myArray objectAtIndex:n-1] valueForKey:@"balance"] floatValue] + [[[myArray objectAtIndex:n] valueForKey:@"amount"]floatValue];
			[[myArray objectAtIndex:n] setValue:[NSNumber numberWithFloat:tThisRecBalance] forKey:@"balance"];
		}
	}
}

It does make you consider, why use ASOC at all, when you can get this kind of speed increase with pure Objective-C?

All the best

Terry

I guess one reason is that in many cases, speed is largely irrelevant – fast enough is fast enough.

Terry,

I couldn’t tell from the above quote, if you thought that my method and your method are doing different things – as far as I can tell, they’re the same (except I used “amountA” instead of “amount” and “amountB” instead of “balance”). I tested your code and mine with a 1 million member array, and they both took around 3 seconds to complete (the fast enumeration method is supposed to be faster than the NSEnumerator method, but doesn’t seem to be faster than the C-style method you used).
That being said, the fast enumeration method I used does give you more compact code and doesn’t require treating the first record differently.

Ric

Hi,

Your right, your method certainly gives more compact code and I am surprised that the various Objective-C methods all run at about the same speed.

What is even more surprising to me is that ASOC is so slow.

I suppose a conclusion would have to be that if you run into a speed bottleneck with ASOC do not bother trying to optimise your code (wasting time like I have) just switch to pure objective-c.

Now another debate would be that if you hit a bottleneck often then don’t bother with ASOC at all just use Cocoa/Objective-c from the outset.

That then leads to a possibility that you are now learning two syntax methods (programming languages I suppose) when the one (Cocoa/Objective-C) can fulfil all your needs?

I have been pondering this situation for sometime. I like scripting languages but ASOC seems to be leading you to learn Cocoa/Objective-C anyway, having to translate all the example code/syntax that is Cocoa/Objective-C to ASOC.

When you have done this perhaps you should just write in Cocoa/Objective-C? and have done with it.
I am not sure.

Does anyone have a story to tell?

All the best

Terry

You were doing a heck of a lot of bridging across languages there – that has to take time.

That’s not an option everyone has, though.

If you’re comfortable with it, why not. But premature optimization is considered one of the great sins of programming.

It’s horses for courses. If you’re app is not actually driving a scriptable app, the only reason to do it in ASObjC rather than Objective-C is familiarity. But if you are driving a scriptable app, you don’t want to be doing that in Objective-C.

Sounds like a nice, gentle introduction…

It’s a choice. And you can’t have too much choice about such things.

Running my AppleScriptObjC code with Xcode4, I’ve encountered this “internal table overflow” AppleScript error Parsing XML data from an NSXMLDocument with PathForNode XPath in a repeat loop. The data is about 128 items with 10 subitems each (1280 records).

Running the ASOC code in Xcode3, I do not get this this error, running the code 100 times. Usually fails within 10-20 times in Xcode4.

The problem I am having now, is my App seems to be growing in Memory. When running the script 10 times, The real Memory starts out at 30MB and grows to 45MB. Garbage Collection is on.

Below is the code involved (it’s not the entire program, but even these two parts are rather lengthly running the code from an NSConnection.)

Is the real memory expansion for ASOC Apps normal? Any suggestions. Would like to avoid redoing everything in Cocoa (Objective-C), but seriously considering it. I did not notice this problem with AppleScript Studio Apps.


--this class is called from the AppDelegate
script channelXML
	property parent : class "NSObject"
	on channelXML_(channelData)
		log "loading new channel list."
		
		set chList to {}
		set myChList to {}
		
		-- this time we will init the XML with Data rather than converting to a string first
		set channelXML to ((current application's NSXMLDocument)'s alloc)'s initWithData_options_error_(channelData, current application's NSXMLDocumentTidyXML, missing value)
		
		set cat to channelXML's nodesForXPath_error_("./lineup-response/lineup/categories", missing value)
		
		log "begin channel repeat loop."
		--log myXML
		
		repeat with wc from 0 to ((count of cat) - 1)
			set workCat to cat's objectAtIndex_(wc)
			set currentCat to stringValue of workCat's elementsForName_("name") as string
			
			log currentCat
			
			set genre to workCat's nodesForXPath_error_("./genres", missing value)
			
			repeat with g from 0 to ((count of genre) - 1)
				set workGenre to genre's objectAtIndex_(g)
				set currentGenre to stringValue of workGenre's elementsForName_("name") as string
				set channel to workGenre's nodesForXPath_error_("./channels", missing value)
				
				log "   " & currentGenre
				
				repeat with c from 0 to ((count of channel) - 1)
					set ch to channel's objectAtIndex_(c)
					
					set isAvailable to stringValue of ch's elementsForName_("isAvailable") as boolean
					
					if isAvailable then
						set isMature to stringValue of ch's elementsForName_("isMature") as boolean
						
						-- Add leading zeros, did it using AppleScript :)
						-- Could not get stringWithFormat to work. :(
						set sirNo to text -3 thru -1 of ("000" & stringValue of ch's elementsForName_("siriusChannelNo")) --as string          
						set xmNo to text -3 thru -1 of ("000" & stringValue of ch's elementsForName_("siriusChannelNo")) --as string
						
						set chName to stringValue of ch's elementsForName_("name") as string
						set chKey to stringValue of ch's elementsForName_("channelKey") as string
						set shortDesc to stringValue of ch's elementsForName_("displayName") as string
						
						set x to stringValue of ch's nodesForXPath_error_("./logos/url", missing value)
						
						set medLogo to item 2 of x as string
						
						log "       " & chName
						
						copy {cat:currentCat, gen:currentGenre, sirNo:sirNo, xmNo:xmNo, chName:chName, isMature:isMature, chKey:chKey, shortDesc:shortDesc} to end of chList
						
						--pgURL_(medLogo, "", getMethod, theDownloadDelegate)
						
						--Download Logos
					end if
				end repeat
			end repeat
		end repeat
		--end
		
		log "finished triple repeat loop"
	
		-- Sort Channel List Array (currently by Sirius Channel Number
		tell current application's NSArray to set unsortedArray to arrayWithArray_(chList) -- make list into array
		
		set arrayDesc to current application's class "NSSortDescriptor"'s sortDescriptorWithKey_ascending_selector_("sirNo", true, "localizedCompare:")
		
		set chList to unsortedArray's sortedArrayUsingDescriptors_({arrayDesc})
		return chList
	end channelXML_
	
end script

Running the ASOC code in Xcode3, I do not get this this error, running the code 100 times. Usually fails within 10-20 times in Xcode4.

Try changing the compiler in Xcode4.

Running something 10 times is no test, especially for 64-bit apps. Memory use goes up and down as required. That doesn’t mean there aren’t any leaks, just that memory use is more complex with 64-bit apps and garbage collection.

After additional tests and while trying to optimized my ASOC code, I’ve determined that working with a very large NSXMLDocument and trying to go through three nested loops will inevitably exceed the memory limit of AppleScript. NSXMLDocument nodes and elements take up a lot of memory which is probably why NSXMLDocument is not on the iPhone.

If I only create one loop through the NSXMLDocument, AppleScript handles it fine, but trying to nest through it two - three times to get additional info that is further up the tree, it seems to be use up too much memory, as many “copies” of objects exist during the nested loops.

I am in the process of rewriting my subclass in Objective-C. Seems to be the best fix for the Internal Table OverFlow and Stack Overflow errors. Another route would be to use System Events to parse the XML data and let System Events handle the parsing/memory overhead. Since I have been wanting to steer away from System Events, a custom Objective-C class seems to be the best answer to my memory problem.

Best news is, I was able to create a prototype in AppleScriptObjC and use it as a blueprint for the subclass.

gt :slight_smile:

Another option would be to use NSXMLParser. It would probably be a fraction slower, but much less memory-hogging.

True. I already had the code figured out using NSXMLDocument and a few XPath queries; another reason to stick with what I had.

Ran into the Table overflow error again with another section of AppleScriptObjC code, downloading 120 images one after another via a 120 separate NSConnections. Crash happens on 2nd or third attempt. Most users won’t be continously redownloading these images, but I think to try to make the code bug proof. This time the code is simple enough to move it back to ObjectiveC. I’m a little annoyed by the memory limits of AppleScriptObjC; you would think Apple could give it some more overhead. Speed is not really an issue for me for ASOC, it’s the table overflow and stack overflow errors. When they happen, thing behave very odd in and can affect unrelated (latter) parts of the code.

BTW, I did get my code to run with or without Garbage Collection. I just had to add an NSAutoRelease pool to main.m. I couldn’t get additional plain ASOC scripts to load as a subclass though w/o GC (seems GC helps ignoring headers), but I am able to get normal ObjectiveC classes to load w/o GC. In my testing I encountered the table overflow error with and without GC, so it is not a problem with taking out the garbage but a inherent limitation of AppleScript. Also when not using GC, dealing with objects like NSMutableArray can be more strick on how they are init’d.

I think the reason why I never found this Table overflow error before (in 13 years) is AppleScript was never taxed enough with AppleScript Studio: used do shell scripts, System Events, or another scriptable application for many operations. AppleScript just did the light lifting.

In my current project, I decided anything that will be CPU or Memory intensive will be an Objective-C class. Anything simpler will be in ASOC (App Delegate, UI, bindings). What I like about having ObjectiveC subclasses, is I can easily move them to a full fledged Cocoa app (when I am ready for that stage). Might just find a happy medium in-between. I still like ASOC. It’s easier and more fun. The GCC compiler for ObjectiveC is strick (but is helpful when you don’t know what you’re doing, it will point out all your mistakes and will sometimes make a suggestion).

Below is my same code converted to ObjectiveC. It was a good exercise for me for mainly being an AppleScript programmer. I quickly learned how to use the ‘For In ObjC 2.0 Fast-Enumeration loop’, did a little bit with booleans, and finally figured how to get StringWithFormat to work (from an NSInteger). :slight_smile: Not much error checking and don’t really know how to setup properties in ObjC yet. Will learn that as I go. Tried to stick with local variables and pointers. At the same time, I did hire an ObjC programmer to do their own conversion and add they added in some extra error checking / options (not included). They did a better job, but was pretty impressed that this didn’t take much time to figure out what I needed. Stumbled through it, but had the right mind set.

Both my ASOC and ObjectiveC versions run at approx. the same speed. (1 second or less to process 120,000 records). The main difference though is CPU. 10% for ObjectiveC. 90% for ASOC. For the same operation. Going back and forth using AppleScript is probably taxing.

BTW, Sticking with Xcode 4.0.2, once I got used to Xcode4, I couldn’t go back to 3.

[code]+ (NSArray *)test:(NSData *)data
{
NSError *error;
NSXMLDocument *document =
[[NSXMLDocument alloc] initWithData:data options:NSXMLDocumentTidyXML error:&error];

NSXMLElement *rootNode = [document rootElement];
//NSLog(@"my rootNode = %@", rootNode);

NSString *catQueryString = 
@"//lineup-response/lineup/categories";

NSString *genreQueryString = 
@"./genres";

NSString *channelQueryString = 
@"./channels";

NSArray *myCategories = [rootNode nodesForXPath:catQueryString error:&error];

NSMutableArray *channels = [[NSMutableArray alloc] initWithCapacity:120]; 

for (NSXMLElement *cNode in myCategories)
    
{
    NSString *myCat = [ [[cNode elementsForName:@"name"] objectAtIndex:0] stringValue];
    // NSLog(@"myCat = %@", myCat);
    
    NSArray *myGenres = [cNode nodesForXPath:genreQueryString error:&error];
    // NSLog(@"myGenres = %@", myGenres);
    
    //NSLog(@"my myCategories = %@", myCategories);
    
    for (NSXMLElement *gNode in myGenres)
    {
        NSString *myGenre = [ [[gNode elementsForName:@"name"] objectAtIndex:0] stringValue];
        // NSLog(@"myGenre = %@", myGenre);
        
        NSArray *myChannels = [gNode nodesForXPath:channelQueryString error:&error];
        //  NSLog(@"myChannels = %@", myChannels);
        
        for (NSXMLElement *myNode in myChannels)
        {
            NSString *isAvailable = [ [[myNode elementsForName:@"isAvailable"] objectAtIndex:0] stringValue];
            
            // NSLog(@"isAvailable = %@", isAvailable);
            
            // Convert isAvailable string to Boolean :)
            BOOL availBool = [isAvailable boolValue];
            
            if (availBool == 1)
            {
                NSString *myName = [ [[myNode elementsForName:@"name"] objectAtIndex:0] stringValue];
                
                NSString *chKey = [ [[myNode elementsForName:@"channelKey"] objectAtIndex:0] stringValue];
                
                NSString *sirNum = [ [[myNode elementsForName:@"siriusChannelNo"] objectAtIndex:0] stringValue];
                
                // add leading zeros to the Channel Number
                NSInteger sirInt = [sirNum intValue];
                NSString *sirNo = [NSString stringWithFormat:@"%03d",sirInt];
                
                NSString *logos = [[[myNode nodesForXPath:@"./logos/url" error:&error]objectAtIndex:1] stringValue];
                
                NSString *shortDesc = [ [[myNode elementsForName:@"displayName"] objectAtIndex:0] stringValue];
                
                [channels addObject:
                 [NSDictionary dictionaryWithObjectsAndKeys:
                  sirNo, @"sirNo",
                  chKey, @"chKey",
                  logos, @"medLogo",
                  shortDesc, @"shortDesc",
                  myName, @"chName",
                  myGenre, @"gen",
                  myCat, @"cat",
                  nil]];
            }                
        }
    } 
    
}

//NSLog(@"channels = %@", channels);
    
[channels release];
[document release];

return channels;

}[/code]