Finding iCal todos fast

Scripting Forums Code Exchange
#1

During my discovery of the neat list of todos which you can print from iCal I also discovered that a whole bunch of my todos didn’t had a due date. Unfortunately ICal prints todos without a due date even if they are completed.
Well, I don’t care about the due date if the todo is completed, so the idea was born to set the due date to the value of the completion date with a small AppleScript. This should be easy, just find every todo with a completion date and a due date with missing value. Then set the due date to the completion date - job done.

Because I have a lot of todos I didn’t want to iterate over each and every todo. There should be a possibility to let iCal do the work of filtering the todos.

To start with something I tried to find all todos without a due date:


tell application "iCal"
	get todos of calendars where due date is missing value
end tell

Result: missing value can’t be converted into a date

Ok, it seems that some more investigation is necessary here. The missing value is somewhat like NULL in SQL, there is no value. The keyword “is” is not like the SQL keyword, it is just a synonym for equals. That means AppleScript has to convert missing value to a date in order to compare both values. It turns out that this is impossible for AppleScript. (That is the same in SQL if you try something = NULL…)

But missing value shares something else with the SQL NULL: If you compare missing value with anything else you get false as the result. (Well not exactly. If you compare missing value with missing value that will give true. So the behaviour is not exactly the same as in SQL.)

Now this piece of knowledge is enough to write a filter which will work. All we need is a date to compare with and which is guaranteed outside our due dates. I picked a date far enough in the past to be sure I had no duties at that time.


tell application "iCal"
	-- at least I had no todos before that date...
	set d to date ("01.01.1000")
		-- find the todos which are not completed
		-- due date = missing value
		set todosToComplete to (todos of Calendar 1 where not (completion date > d))
		-- for these todos set the due date to completion date
		repeat with theToDo in todosToComplete
			copy {ident:uid, prio:priority, descr:summary} of theToDo to the end of myTodos
		end repeat
	myTodos
end tell

The tricky part is “not (completion date > d)”. If you just say “completion date > d” you will get all todos completed after our arbitrary start date but no todo with a completion date with missing value. To get every todo with a completion date the date we compare to has simply to be far enough in the past. Now we get all completed todos. To get the ones which are not completed we simply negate the condition with “not”.

Ok, ready for the solution of my task to set all due dates to the completion dates if the due date is missing and the completion date is set.


tell application "iCal"
	-- at least I had no todos before that date...
	set d to date ("01.01.1000")
	-- for each calendar
	repeat with theCal in calendars
		-- find the todos which are completed but have no due date
		-- due date = missing value and completion date <> missing value
		set todosWithMissingDueDate to (todos of theCal where not (due date > d) and (completion date > d))
		-- for these todos set the due date to completion date
		repeat with theToDo in todosWithMissingDueDate
			tell theToDo to set due date to completion date
		end repeat
	end repeat
end tell

No need for iterating over every todo anymore!
Happy Scripting!

Model: MacBook Pro
AppleScript: AppleScript 2.1.2
Browser: Safari 534.55.3
Operating System: Mac OS X (10.6)

#2

Hi. Welcome to MacScripter and thanks for posting your solution!

Notice that the full error message is: “iCal got an error: Can’t make missing value into type date.” It’s not actually AppleScript doing the converting and comparing, but iCal. AppleScript only gives iCal the instructions and receives the results, if any. If AppleScript asks iCal for the due date of a todo which doesn’t have a due date, iCal returns an AppleScript ‘missing value’ object to keep AppleScript happy. But it’s very unlikely that it uses such objects internally. It’s also unlikely to use AppleScript date objects for date values, so when AppleScript passes it a reference like ‘where due date is date (“01.01.1000”)’, iCal has to convert the date object to the format it uses internally. It knows how to do this, but is flummoxed if the reference contains ‘missing value’ instead of a date. Your script passes a reference that iCal knows how to process.

It’s logically fascinating. ‘todos where not (completion date > d)’ returns a list of todos which don’t have completion dates, whereas ‘todos where (completion date is not greater than d)’ returns an empty list! In the first case, iCal’s thinking of todos not having a completion date greater than d, which includes not having a completion date at all. In the second, it’s thinking of todos having a completion date, but one not greater than d.

I find with most scriptable applications that applying a ‘whose’ filter to a large pool of objects can be quite slow, even when only a small number of the objects is actually returned. It’s often faster to get the application to return the relevant information for all the objects in the pool without thinking about it and to let AppleScript do the filtering. This also allows AppleScript to sift through the data itself without additionally having to ask the appliction to return them for each individual returned object. Applied to your middle script, such a process would look something like this:


set myTodos to {}

tell application "iCal"
	-- Get parallel lists of relevant properties for all the todos in the calendar.
	set {complDates, idents, prios, descrs} to {completion date, uid, priority, summary} of todos of calendar 1
	-- If any of the returned items in the completion date list are 'missing values', collect the equivalent items from the other lists.
	repeat with i from 1 to (count complDates)
		if (item i of complDates is missing value) then set end of myTodos to {ident:item i of idents, prio:item i of prios, descr:item i of descrs}
	end repeat
	myTodos
end tell

. and to your final solution:


tell application "iCal"
	-- Get parallel lists of relevant properties for all the todos in the calendar.
	set {dueDates, complDates, idents} to {due date, completion date, uid} of todos of calendar 1
	-- If any of the returned due dates are 'missing values', collect the equivalent items from the other lists.
	repeat with i from 1 to (count dueDates)
		if (item i of dueDates is missing value) then set due date of todo id (item i of idents) of calendar 1 to (item i of complDates)
	end repeat
end tell

I don’t have enough iCal todos to compare the AppleScript and iCal filtering speeds here, but you may find the technque interesting. With very long lists, referencing techniques can be used to speed up access to the values.

#3

Thanks for the warm welcome!

True. Thanks for the much more correct explanation.

I’am not sure about that…
iCal uses a sqlite3 database as the backend to store its events and todos. If you take a look into that database you will find that due date, completion date and so on are stored as timestamps. At least this is a datatype for storing dates.

(You can try this in a Terminal window with:

sqlite3 ~/Library/Calendars/Calendar\ Cache
sqlite> .schema ZCALENDARITEM
Also try to select all todos (I believe z_ent = 9 marks all todos, but I am not sure about that) without a completion date:
sqlite> select ztitle, ZCOMPLETEDDATE from ZCALENDARITEM where z_ent = 9 and ZCOMPLETEDDATE is null;
)

Try to think about everything after where as an boolean expression which is evaluated for each todo.

For an todo without a completion date:

  1. evaluation step
    completion date from todo → missing value
  2. evaluation step, because of the round brackets
    missing value > date(“01.01.1000”) → false (because everything compared to missing value results in false)
  3. evaluation step
    not false → true → included in result set

For an todo with a completion date:

  1. evaluation step
    completion date from todo → date(“08.05.2012”)
  2. evaluation step, because of the round brackets
    date(“08.05.2012”) > date(“01.01.1000”) → true
  3. evaluation step
    not true → false → not included in the result set

Now compare that to your second expression “completion date is not greater than d”
For an todo without a completion date:

  1. step
    completion date from todo → missing value
  2. step
    missing value <= date(“01.01.1000”) → false → not included in result set

For an todo with a completion date

  1. step
    completion date from todo → date(“08.05.2012”)
  2. step
    date(“08.05.2012”) <= date(“01.01.1000”) → false → not included in result set

For me that sounds absolut logically :).

I agree. I’ve found this very true more than once. But it highly depends on the particular application and the actual filter. Because iCal is backed up by sqlite3 I assume that the whose filter will be evaluated by sqlite3, which should be reasonable fast.

Performance might be better with your approach, depending of how many todos you have in total and how many are without a due date (or whatever the test is). It will be mainly influenced by how many Interprocess communication calls are necessary.
Your approach always needs so many calls as properties which you retrieve. My approach uses fields * todos found + 1 (for the filter evaluation) number of calls. The second variable seems to be the number of repeat loops and the involved tests for filtering the todos in AppleScript.

Last but not least I find the whose filter and iteration over todos more readable then working with parallel lists, but your mileage may vary.

Neverless, thank you very much for inspiring input.

#4

I’ll add my welcome to Nigel’s, sroeper, and thank you both for a wonderful exchange. This is exactly what Code Exchange was meant to encourage. Knowing Nigel, he’ll be studying your commentary in the light of his.

#5

This might help: when you pass an AS date for a property that expects a date, it gets converted and passed to the application as a Cocoa date object (NSDate). The application then presumably does things with that date. But missing value in this case gets translated to nil, not to a null object.

The thing with an object set to nil in Objective-C (which is what iCal is using) is that when you call any method on it, the result will also be nil. So typically Objective-C code checks to see if a variable is nil, and if so doesn’t go any further because there’s no point – it’s not an object and it can’t be used in comparisons or anything else.

#6

Nigel’s comment about the performance is entirely right, I assumed to much about the inner workings of iCal…

iCal seems to be not any better in evaluating the result of a ‘whose’ filter then most other applications. I tried to confirm that it is better by building up a test last night which creates a new calendar with 1000 todos, 88 of them didn’t had a due date another 88 had a missing completion date.

My expectation was that a big pool and a small result set paired with a complicated test will have an positive impact for the 'whose". The result is sobering. ICal went more than once into a not responding state. Only with a relative small set of todos (less than 100) and an even smaller result (preferable 0) I was able to be faster with ‘whose’.

My conclusion: Avoid ‘whose’ if you can, at least do some serious tests if performance matter

For everyone whose interessted, this is the better solution (thank’s to Nigel):


tell application "iCal"
	-- for each calendar
	repeat with nc from 1 to (count calendars)
		set theCal to item nc of calendars
		-- Get parallel lists of relevant properties for all the todos in the calendar.
		set {dueDates, complDates, idents} to {due date, completion date, uid} of todos of theCal
		-- If any of the returned due dates are 'missing values' but the equivalent 'completion dates' are not, set the due date to the completion date
		repeat with i from 1 to (count complDates)
			if (item i of dueDates is missing value) and ¬
				(item i of complDates is not missing value) then
				set due date of todo id (item i of idents) of theCal to ¬
					(item i of complDates)
			end if
		end repeat
	end repeat
end tell
#7

Don’t jump to conclusions on the basis of one app. In some apps, the advantage is massively the other way.

#8

I was talking about iCal. Sorry if this point isn’t clear enough. And for any other app, which I haven’t written by my self, I will do some tests before I will use whose. (If I concern about running time and stability).