Trouble with internal table overflow in Script Debugger 4.5.7

McUsr · July 25, 2012, 5:58pm

Hello.

I have trouble with this a lot of times, usually, I just hit enter in my script, and recompiles. But now, I have a script with 2600 lines, and it isn’t that easy all the sudden.

I have some questions about remedies for the internal table. I have read this post: Script has grown too large for editors.

Does it help to encapsulate globals in a script object?
will it help to have globals stored into lists, to save declarations?
Will it help to encapsulate larger parts of the script into script modules within it, and will this slow my script down? … (I need all the speed I can get.
Does number of subroutines count?
Can bugs ” uncatched lead to this?

I do have script objects in my cody, that I copy into globals, to ensure that I start afresh whenever the script is run, is that a bad practice?

*Does it help to replace literal strinconstans with a variable, containing the same constant. (I.e is the compiler smart enough to replace every instance of a string constant with one of them, or are they looked upon as individual entities?

The string constansts are permeated through my code. I have three globals defined which are having this structure copied into it at run time:

script dataItem 
	property raw : {} 
	property sorted : {}
	property reltable : {}
	property inv_reltable : {}
	property display_list : {}
	property isDirty : true
	property dsp_offset : 0 as integer
end script

The properties are used liberally through more subroutines than not, and those not having them as globals, those takes the them as parameters.

I also have a great number o top level properties in my script, for localization. And, there isn’t really that much code to eventually dump out after initialization, and the code are 100% factored to subroutines, by tomorrow, as I have three left, and three lines of code to replace with a handler.

I have questions about the internal table, knowing nothing about it. for instance; if I encapsulate the strings into a script object, won’t that lead to even more table references?

Is this internal structure, is it a flat table, or is it nested? Does it only consist of objects, or is it used to dereference?

I addition to this, and what is said in the post above, I hope you tell me which questions, I really should have asked!

Thanks

mouramartins · July 25, 2012, 10:12pm

Try this, if possible:

script dataItem 

  property parent : Applescript

   property raw : {} 
   property sorted : {}
   property reltable : {}
   property inv_reltable : {}
   property display_list : {}
   property isDirty : true
   property dsp_offset : 0 as integer
end script

Shane_Stanley · July 26, 2012, 2:24am

It should, yes.

Nothing is slower than a script that won’t compile or run.

It’s an implementation detail – all you really need to know is that there are limits to everything, including script lengths and nesting levels. The limits may (and probably do) vary with OS versions, as may how the internal table is used and implemented. The safest thing to do is be conservative: when a script gets too big, start breaking it into smaller bits.

McUsr · July 26, 2012, 4:09am

Hello!

Thanks for your input Shane!

Well! I loose nothing, encapsulating it, but a very small fraction of a second, and shouldn’t work, then I have a script object I can factor out during development anyway. Script Debugger will stil let me see the code in the script object execute, and what I am really after is making it compile in Script Debugger, so I can use the facilities.

This has seemed to been a Script Debugger thing, during debugging. So I think I can manage to assemble it back into something working afterwards.

DJ_Bazzie_Wazzie · July 26, 2012, 11:07am

I never worked with script debugger but what Shane said applies for all scripting and oop languages; including script debugger. Script debugger is just an IDE but like every IDE, it has to respect the limitations of the programming language and runtime limits (if there is any). There is a legion of limits in OOP (like) langauges like maximum string length of a handler name, maximum arguments for a handler, depth of iterations, nesting level of operators, the size of an object/class, the amount of handlers in a class, the amount of properties in a class, the amount of globals (stored in the root table, no matter where the declaration is) and nesting levels of classes.

Consider this: When you create a language like AS you consider many different situations and think: ‘if the user want to exceed these limits then there is definitely something wrong’. What I’m trying to say is that when you bounce against the limits of a programming/scripting language your using it in a way it isn’t designed for and can be considered as ‘bad’ programming. For instance it is not allowed to have more than 4096 members in a C++ class; this number is chosen as a will-never-happen-situation.

The following comments can be considered as speculative, assumptions and therefore can’t be considered as facts:
As far as I know the stack is hierarchical, I’m not sure for tables. It’s assumable that every script object has it’s own table and there is only a single stack (at runtime). That the stack is a hierarchical structure containing tables could be possible as well. That means a table represents an object and listed all handler names, events, properties, class name etc… just like a C struct represents an Objective-C or C++ class/object. I’m sure I’m not far away from what actually happens but only Apple knows the inside of the interpreter.

Interesting:D

McUsr · July 26, 2012, 11:34am

Hello!

First of all, I use deliberately as few of the OOP features as I can get buy with, as that is surely something that adds up to that table quickly, and also slows down the code.

It is beautiful, but not suited for what I am doing, so I have writing totally procedurally, but I still meet the wall.

It would have been interested to know how the internal table is constructed, and there are some posts here, and other places discussing it, why there is limits, and how they come into daylight. Memory usage being one of them.

I think however I have understood, that there is one such table, created for each object, so I will circumvent it, by partioning out code while developing. Script Debugger lets you trace through code that is loaded as an object anyway!

The clue is, is that it has happened me far to early really, as there is nothing big, about my script in anway, so I supsect that I do something stupid, in my code somewhere, that creates the hickup.

Something that doesn’t necessarily is a blatan’t error, but something that leads to over using the internal table. Maybe as subtle as having to many tids statements for all I know. maybe some calls to Osaxen’s that should have been formed differently.

The only thing I think is efficient to do, is to circumvent it by partioning the code into units, that lets me continue to develop with the benefits of ScriptDebugger where it matters.

If I could only buy one piece of Software… od -cb is just one of the things it is, you are literally a blind man without it! You can go in and look at the internal structures, while a scriptable app is running!

One more minor detail … when you fire up a script with Quicksilver, and it is saved in debug mode, then it will be opened in script debugger at once, now how cool isn’t that! (Just adding it here in my praise! )

Shane_Stanley · July 26, 2012, 12:09pm

And I wouldn’t touch AppleScript without it:P

To be clear, though: I think what McUsr is seeing is an AppleScript table overflow. Script Debugger can sometimes trigger them a bit more easily in debugging mode because it obviously has to include extra code for the debugging to work, but in my experience it doesn’t lower the ceiling much at all.

I reckon 2600 lines is a very long AppleScript script – I start getting twitchy around the 600-line mark.

McUsr · July 26, 2012, 3:44pm

Hello!

It runs pretty well when not in debugging, at the moment I guess I have to add another maybe up to 700 lines or so, (doubling my esitmate not having scripted it yet), and when I do that, then I might like to have it working in ScriptDebugger for the assembly.

But I don’t need to, as this luckily happened after the assembly of the core so to speak.

Now I have removed some 50 lines by optimization (which there is always good to have a reasonably excuse to do, and this is it) , I have factored out some 100-200 lines into a module, and it still trips, with less lines than where it got grumpy originally.

I have some handlers that just are copy pasted with some minor changes in the loop, I suspect that may be the one.

I have also made a maker function for my dataItem, so it doesn’t float around loose, I am also pondering setting the parent to Applescript like mouramartins suggested.

I am also declaring variables local

Edit

I am also working through the tips Mark Alldritt comes with in this article.

Thanks for your input.

McUsr · July 27, 2012, 6:27am

Hello!

I have managed to remove like 200-300 lines of code - 80k of compiled script! This problem is intriguing. I not seeing the usage of libraries to help much really, if used conventionally, but if I use them, like I am sending messages, like:


tell libUtil to myhanlder(param1, param2)

As a way to circumvent the table overflow. Because that table is there to connect the dots runtime so to speak right?

So whether I have globals, or I have parameters, everything will get into there. But! If I send messages, now that would be like to some external application I guess ( I have also seen that as a cure for similiar problmes in mailing lists.

I have considered the usage of AppleScirpt as a parent property, and figuring out how to message those, but the the thing is, that that only helps so much, as I haven’t got any real objects, but only prodedural code in the first place.

I have also used a lot of copy statements. Those perform deep copy, maybe I have been careless with some of those!

Anyway, I have figured out a shallow copy handler it goes like this:


to shallowCopy from something
	local newThing
	set newThing to contents of something
	return newThing
end shallowCopy

That should just make another instance of an object, it doens’t give any sense when copying scalars, but
this is an area where I am paranoid, having dealt with closures before. I think I may have copied something, and gotten a little more than I bargained for, that is at least something I will look into before I start to redoing code! like if I get an implicit parent property if I copy a script object! loathe the thouhgt of having to restructure, when I finally have it working! (except for some small easily fixed bugs I have introduced lately)

mouramartins · July 27, 2012, 6:40am

I had a similar problem while using script objects to parse the tags in AAC files. The whole file, like QuickTime movies is structured as nested objects.

I believe ScriptDebugger needs some kind of copy of the whole script everytime a copy of the script object is made. With a threat of infinite recursion. After all that’s what you can see in the parent property of the variable: a copy of the whole script, including the script object with it’s default values for properties.

The script is big like in 1000 lines, making heavy use of OOP features and started choking with internal table overflows about halfway.

The parent : Applescript thing solved the problem once and for all, placed just at the object in the top of the hierarchy. Since then I adopted the explicit use of the parent property: daddy is cool!

McUsr · July 27, 2012, 7:10am

Hello!

Well, my top level object, is really my script file, I’ll try that first, maybe.

And thanks for the clarification, as to why

Edit
That did not go so well. But I’ll certainly have it in the back of my head, as I am going through the copying of script objets, and give it a serious try, if it kicks in, then I certainly know where the dog is buried!

The problem is that it can be so many things, it can for instance be too many colours used in the Applescript formatting!

Honestly, this really happens, before it should have happened before the script really is up and running, so I just tried it now, for having done so, and…


say "[[rate 350]]OMG. OH[[rate 350]] YE[[rate 250]]S IT WORKED!" using {"Bruce"}

What more is there to say? I tried it, it worked

Thanks a whole Bunch!

You really don’t have use Script Debugger for getting in trouble with this, but maybe it is a benefit really, that you can’t have such problems lying around latent in your code when using Script Debugger, as Script Debugger needs to take care of debug information, and that adds up quickly when things are copied deeply, taking the whole environment with it.

I think Smile, at least was the most liberal, with concern to memory, and things go generally good, until you get it up in your face. I prefer getting it in my face up front.

Thanks a lot again!


tell application "Quicksilver" to show large type "Thank You Very Much!"

I wasn’t thinking in those orbits for starters, so now I am back on track, with optimized code!

Edit II
The funny thing is, is that now it works, but putting in that property, didn’t make the executable with debug info any smaller, on the contrary, I think it grew! But, it works! I guess it copies in Applescripts environment for each and every object, I am really thinking of applying an “object maker handler” like this.


to makeObject()
script blah
property bla : missing value
property more : {}
end script
end makeObject

And copy properties at a later state, one by one, with the shallowcopy handler, to see if that makes the script both smaller, and if that patent works. I’ll report back on it.

I may even have three similiar script objects at the top level in my scirpt that I “rinse” every time from the run handler, before I assingn them to their respective globals, if my first attempt fails.

mouramartins · July 27, 2012, 11:43am

McUsr · July 27, 2012, 1:11pm

Hello!

There is no way I am going to make this work without setting the parent property. I tried, avoiding copy operations, perfoming shallow copies of properties, to no avail!

Moyramartins adds to the script size, like adding a script object to a handler, on a per object basis. Whether I set the parent to Applescript or integer, it doesn’t work! I go for Applescript, since that seems most natural.

That increased size doesn’t matter, as long as Script Debugger still works!

Thanks again!

McUsr · July 28, 2012, 4:18am

Hello!

There are still issues with my way of coding, that I try to improve.

A global is per se a property, which means that it may take up storage on disk, so I have found a way to be sure that it is redeclared everytime, which is somewhat cheaper than to call a maker function and all that. And also ensure that globals, that will be redeclared, wont’t be stored to disk by undefining them before exiting, so there is nothing to store, leaving the script in a saved state smaller.

Globals that are merely used as constants throughout code, will be promoted to real properties, those doesn’t change value anyway.

The script is starting to be big, which means that it will take to load, so I want to make it up and running as fast as possible, the less compiled code there is, the faster it will start. For the same reasons, I want as few handler calls as possible, wanting the script to show results asap.

This is the way I will make my volatile globals: the first statmenent in the snippet below, (together with the size, is just to prove that nothing got stored on disk!, try commenting out the undef statement, and check the size between runs, before enabling it again.

Organizing it this way also leaves me with the benefits of being able to reference my variables, with the my keyword, with the speed improvements therein. Practically leaving me without the need for neither accessing the top level property throughout my code via a global declaration.
nor do I need to have a script object within handlers to speed up access to elements during loops, when starting derefencing the list items with the my keyword. This omittance of the various script objects within handlers, will also save somewhat to the overall size of the script. (This will of course not hold for general handlers, that works on any list, those will stil need their own script object to speed things up. I also believe I will benefit from getting acccess to my structurs by the global keyword, when I have no intention of accessing its members. )


on run
	--------------- testing
	try
		if my c's b > 0 then log my c's b
	end try
	--------------- declaring 
	script a
		property parent : AppleScript
		property b : 5
	end script
	global c, d, e
	--------------- making 
	copy a to c
	copy a to d
	set e to a
	--------------- using 
	log my c's b
	b()
	log my c's b
	-------------- undefining 
	set { d,e,f } to {undef(),undef(),undef()}
end run

to undef()
	return
end undef
to b()
	set my c's b to (my c's b) + 1
end b

The last little tweak of speed, will ofcourse be to avoid the copy command all together, but declare the number of alike script objects I need in the run handler by different names, and assign them to their respective global. That is another experiment, as I fear the compiler may see that they are alike, and outsmart me, by simplyfying the code, making it into just one structure.

The compiler does indeed not coalesce identical structures into one:
This adds of course to the overall size of a script, but on the other hand, I think getting it by declararation this way is faster, than either using an “object-maker” handler, or the copy command, especially when it comes to script objects!

on run
	--------------- testing
	try
		if my d's h > 0 then log my d's h
	end try
	
	--------------- declaring 
	script a
		property parent : AppleScript
		property h : 5
	end script
	script b
		property parent : AppleScript
		property h : 5
	end script
	script c
		property parent : AppleScript
		property h : 5
	end script
	global d, e, f
	
	--------------- making 
	set {d, e, f} to {a, b, c}
	
	--------------- using
	log "d : " & my d's h
	g()
	--------------- checking if the script objects are alike
	log "d : " & my d's h
	log "e : " & my e's h
	log "f : " & my f's h
	-------------- undefining 
	set { d,e,f } to {undef(),undef(),undef()}
	
end run

to undef()
	return
end undef
to g()
	set my d's h to (my d's h) + 1
end g

McUsr · July 28, 2012, 6:20am

Hello!

Another thing that I think may help me, when keeping the size of the internal table down, while at the same time gaining maximum flexibility, is when I factor out code.

The reason for factoring out code, is obvious, to have only one piece of code, that does one thing, minimizing the places you have to change stuff, and minimizing the number of bugfixes.

The stuff that you can factor right out, is of course easy. When the code you wish to factor out, is in the middle of a handler, not so, when you wish to factor out the code that embellish a handler, but it is fairly easy, as we do have kind of function pointers in AppleScript

There are many ways to accomplish this, but in my case there are really only two alternatives, it is either using a script object, or a global to hold the handler sent into the handler with all the embellishing code.

I don’t believe globals in this case, will lead to any unforeseen problems, regarding closures, but it will add a global to the table size, so the way I choose to provide for a way to run a handler by passing it to another, centralizing the embellishing code in one place is like this, where the redundant code is presumed to be factored into the dothis() and dontdothis() handlers, and the really divergent gode are factored into the myhandler() handler.


on myhandler(stuff)
	display dialog stuff
	return 5
end myhandler

on dothis(what, aParam)
	script whatToDo
		property mhandler : what
	end script
	
	whatToDo's mhandler(aParam)
	display dialog result
end dothis

on run
        dothis(myhandler, "hi")
end run

And not like this:


on myhandler(stuff)
	display dialog stuff
	return 5
end myhandler

on dontdothis(what, aParam)
	global whatToDo
	set whatToDo to what
	
	whatToDo(aParam)
	display dialog result
end dothis2

on run
        dontdothis(myhandler, "hi")
end run

Because in the first example, everything should really be kept in place, maybe it shows up in the internal table for a limited period of time, but is should pass away again, not add to the static size, this is of course just speculations, but it is an effort to try to find a best practice for the future. (I have been lazy, and used the second construct). It is also a better over all practice, since you then don’t have unnecessary globals scurrying around in your code.

McUsr · July 28, 2012, 9:12am

The ShallowCopy was an extremely bad idea.

I’ll leave this to people that understand such matters.

I copied a list from one property of one script object to one property of another script object. Seeing the handler you would believe you would get a copy, but without any overhead from the copy command.

But it just doesn’t work like that, it actually, how strange it must seem, delivered a reference to the property of the first script object to the second!

The offending handler:


to shallowCopy from something
	local newThing
	set newThing to contents of something
	return newThing
end shallowCopy

Not that I had any use for this construct anyway, now that I use the parent property

McUsr · July 30, 2012, 6:28am

Hello I have concluded the matter.

Things work perfect now, but when it breaks again, for other reasons, then I think I have reached the limit.
I surveyed the topic of internal table overflow a little, and here is some useful links I found!

The problem that copy copies the paren too

Good thread describing different problems

Here is another one, where also the parent property is what solves it.

This one takes up problems with really large scripts.

I also came by something very legible by haas where he states that one should use message passing to libraries, as a way to avoid internal table overflow, I have lost the url, but that sounds very legible to me.

Your script doesn’t need to keep track of stuff if it is sent out as messages.

McUsr · July 30, 2012, 12:34pm

Hello.

I also found this article helpful to understand what went wrong with my shallowcopy attempt. This is an article containing some indepth details, when it comes to the container type storage classes, but doesn’t cover the script object.

Where it really shines, is when it comes to shallowcopy and the article contains a well of useful explanatory diagrams, as to what goes on during the different kind of list operations.

It can be found here.
There is source code to go with the articles, which is found in a link inside, the sourcode contains a “unit-test” framework, that may be useful for the developer of libraries.

DJ_Bazzie_Wazzie · July 30, 2012, 11:50pm

Globals can never behave as properties and visa versa. Globals are stored in the Stack and not stored next to a script objects like properties do.

have you measured the speed of calling handlers? AS can do hundreds of thousands in less than a second. So start looking if you have loops, and speed them up? Do you use filters as much as possible (remove unnecessary events)? Can be a shell with piped code replace several commands (speed things up).

My is a script object like any other and therefore in some cases a shorter way but not always. Another way of still keeping the speed in lists using an script object and using the my (where my can be different when following the parenting chain). This is what I like about AS:


script sharedFastList --singleton; when stored in a file remove script and end script line
	on newList()
		script fastListInstance
			property l : {}
		end script
	end newList
	
	on indexOfObject(o)
		set x to 1
		repeat with i in my l
			if contents of i is o then return x
			set x to x + 1
		end repeat
		return 0
	end indexOfObject
	
	on addObject(o)
		set end of my l to o
	end addObject
end script

set myList to newList() of sharedFastList

repeat with x from 1 to 12500
	tell myList to addObject(x)
end repeat

indexOfObject(12498) of myList

The code above also shows no speed difference because it’s inside a handler in a script object that calling the property directly. The ‘end of’ command makes everything slower.

I would say you’re not make full advantage of the parenting chain in AS’. There is no (OOP) compiler or interpreter that can solve this issue. We have a parenting chain and you should stick with that. If one script object seems similar to another you only have to write the differences between these two. Here an example of how extensions can be worked out.

script A
	property x : "a"
	property y : "b"
	
	on handler1()
		return x
	end handler1
	
	on handler2()
		return my y
	end handler2
end script

script B -- an extension of A
	property x : "c"
	property y : "d"
	
	property parent : A
	
	on handler3()
		return x
	end handler3
	
	on handler4()
		return my y
	end handler4
end script

return {A's handler1(), A's handler2(), B's handler1(), B's handler2(), B's handler3(), B's handler4()}

As you can see I’ve use to call handler2() from script A though script B with my first example in this post. But as you can see you can do more.

McUsr · July 31, 2012, 3:54am

Now I wouldn’t be so categoric about that, first of all, there is the exception of things declared in the implicit and excplicit run handler.

I am pretty sure I have read that Matt Neuburg at some page in Apple Script the Definitive Guide states that a global is a property.

Run the code below. Save your script, close it, Open it, and you should start off from the next higher number, so globals are stored on disc, just like properties! I think it differs in that you can’t use the a reference to keyword, But it shouldn’t differ in much more when it comes in differences between a global and a property.


to reader()
	global a
	log a
end reader

to writer()
	global a
	try
		set a to a + 1
	on error
		set a to 0
	end try
end writer

on run
	writer()
	reader()
end run

I like the polymorphism built into AppleScript, but having such things resolve, may take additional time. It may very well be that I can make such a solution, more elegant not with that big speed overhead, but right now, I have no time for that, but I may look into it at a later stage, I don’t suspect such a solution to be shorter or faster, and, I think resolving of references my add to the internal table size.

And the trick for using the parent chain is also setting AppleScript as the top parent, to avoid the saving of everything in the objects scope, when copying. Just saying!

Thank you for your interest, it is interesting!

Edit I’ll actually look into the use of objects, as it may make some parts of my code reusable. The main part of my script is a more than simple finite state machine, and will not gain anything from using OOP on it, as such. The datastructures may be alike, but the rules for acting on them differ to significantly, than it can gain anything from oop, only sharing 4 or 6 common low level handlers, 2 of which are consolidated to take care of different cases, in order to centralize the details into one place.