Sunday, November 19, 2017

#1 2014-08-23 01:23:57 pm

treed
Member
Registered: 2014-04-26
Posts: 21

Parsing JSON files

I need to be able to read in a JSON file (no big deal) and then parse it within AppleScript. I can't use third-party faceless apps and the like, because my script needs to be self-contained and easy for inexperienced users to use.

Does anyone know of any kind of open-source JSON parser built in native AppleScript code? If not, I will need to write a basic one myself... any tips on the best string manipulation routines in AppleScript to do this sort of thing? I don't know a lot about AppleScript's capabilities in this area.

Thanks in advance!

Offline

 

#2 2014-08-23 01:47:49 pm

McUsrII
Member
Registered: 2012-11-21
Posts: 3046
Website

Re: Parsing JSON files

Hello.


There is a */json/tool.py file for parsing json with python, that should be in your python installment.

If you click forums, you'll find a topic called string handling. There is also a text item delimiter tutorial in the unscripted forum.

Good luck! smile


Filed under: JSON

Offline

 

#3 2014-08-23 01:52:12 pm

StefanK
Member
From:: St. Gallen, Switzerland
Registered: 2006-10-21
Posts: 11482
Website

Re: Parsing JSON files

Hi,

it's a pity that you don't want to use at least a AppleScriptObjC script library.
Cocoa can convert JSON to array/list or dictionary/record in one line.


regards

Stefan

Offline

 

#4 2014-08-23 03:43:22 pm

Yvan Koenig
Member
Registered: 2006-09-14
Posts: 3201

Re: Parsing JSON files

I would use a script stored as a package.
In the package would be :
(1) the script itself
(2) the library file.

The script would check if the library file is already installed in the Script Libraries folder.
If it isn’t, it would copy the embedded version to the named folder.
After that, in both cases, it would do its "normal" duty which means call the library to decipher the json file.

Yvan KOENIG (VALLAURIS, France) samedi 23 août 2014 22:43:17

Offline

 

#5 2014-08-23 06:58:17 pm

DJ Bazzie Wazzie
Member
From:: the Netherlands
Registered: 2004-10-20
Posts: 2724
Website

Re: Parsing JSON files

treed wrote:

I need to be able to read in a JSON file (no big deal) and then parse it within AppleScript. I can't use third-party faceless apps and the like, because my script needs to be self-contained and easy for inexperienced users to use.

Does anyone know of any kind of open-source JSON parser built in native AppleScript code? If not, I will need to write a basic one myself... any tips on the best string manipulation routines in AppleScript to do this sort of thing? I don't know a lot about AppleScript's capabilities in this area.


The closest you can get is using bash commands like Python or AppleScriptObjC. There are two problems with parsing JSON. The first is that associative arrays/objects are not as dynamic interchangeable with AppleScript because records with user defined fields are not as easy to use as in JavaScript. You can't create records on the fly for example without using cumbersome methods (store them in a file and print them using osascript). This is the real problem you're facing in AppleScript and only AppleScriptObjC, faceless background apps like AppleScriptObjC runner made by Shane, 3rd party scripting additions and Bash are the only things I can think of that can help you manage those records. The second is performance, JSON is a perfect way to parse while reading (streaming) which makes it very useful even outside JavaScript because it's faster to marshal and unmarshal than XML for example. AppleScript is not designed to do this, so even normal data could end up in slow code while a python command or AppleScriptObjC solution would parse and get the right data out of it in a blink of an eye.

Here an python example which can save you a lot time in AppleScript:

Applescript:


set JSONString to "[{
\"name\": \"AppleScript\",
\"type\": \"Scripting Language\",
\"see also\": [ \"Hypercard\"]
},
{
\"name\": \"JavaScript\",
\"type\": \"Prototype Based\",
\"see also\": [\"ECMAScript\", \"Python\"]
}
]"

do shell script "python -c 'import json,sys
obj=json.load(sys.stdin)
print obj[1][\"type\"]' <<<"
& quoted form of JSONString

Offline

 

#6 2017-09-10 06:02:53 pm

bmose
Member
From:: Massachusetts
Registered: 2006-01-03
Posts: 219

Re: Parsing JSON files

DJ Bazzie Wazzie wrote:

The closest you can get is using bash commands like Python or AppleScriptObjC.

This is an old post that I'd like to comment on.

The following compound command transforms a JSON string into an equivalent Applescript value by taking advantage of the remarkable similarity of the Applescript and JSON value specifications. The primary differences between the two seem to be JSON's use of double-quoted text strings for record labels (called object keys in JSON) and JSON's allowance of a single value to span multiple lines.  JSON's use of square brackets to enclose lists (called arrays in JSON) does not require transformation, since square brackets are an acceptable alternative to curly braces for enclosing Applescript lists. The sed command converts double-quoted object keys to piped record labels (which "automatically" become unpiped if not needed), and the tr command strips any linefeed or carriage return characters:

Applescript:

set applescriptValue to run script (do shell script ("echo " & jsonString's quoted form & " | sed -E 's/\"([^\"]+)\"[[:space:]]*:[[:space:]]*/|\\1|:/g' | tr -d '\\n\\r'"))

Using an example taken from the web:

Applescript:


set jsonString to "{
\"MenuID\":5,
\"MenuVersion\":1,
\"MenuName\":\"Lunch Menu\",
\"MenuItems\":[
{
\"Name\":\"TUSCANI MEDITERRANEAN CON POLLO\",
\"Description\":\"Pasta\",
\"PKID\":2,
\"ParentID\":1,
\"Ingredients\":[
{
\"PKID\":123,
\"IngName\":\"Cheese\",
\"Included\":true,
\"ExtraPrice\":0
},
{
\"PKID\":124,
\"IngName\":\"Sausage\",
\"Included\":false,
\"ExtraPrice\":0.99
}
],
\"ItemPricing\":[
{
\"PKID\":456,
\"SizeName\":\"Large\",
\"SizePrice\":12.99
},
{
\"PKID\":678,
\"SizeName\":\"Small\",
\"SizePrice\":14.99
}
]
}
]
}"


set applescriptValue to run script (do shell script ("echo " & jsonString's quoted form & " | sed -E 's/\"([^\"]+)\"[[:space:]]*:[[:space:]]*/|\\1|:/g' | tr -d '\\n\\r'"))
-->
{MenuID:5, MenuVersion:1, MenuName:"Lunch Menu", MenuItems:{{|name|:"TUSCANI MEDITERRANEAN CON POLLO", |description|:"Pasta", PKID:2, ParentID:1, Ingredients:{{PKID:123, IngName:"Cheese", Included:true, ExtraPrice:0}, {PKID:124, IngName:"Sausage", Included:false, ExtraPrice:0.99}}, ItemPricing:{{PKID:456, SizeName:"Large", SizePrice:12.99}, {PKID:678, SizeName:"Small", SizePrice:14.99}}}}}

It has successfully decoded a number of JSON strings in my testing thus far. Are there any flaws with this straightforward decoding technique? (Two that come to mind are: (1) if a JSON text string value contains a pattern mimicking an object key, i.e., a double-quoted string following by a colon character, presumably a fairly rare occurrence in real-life usage; and (2) if a JSON text string value contains a linefeed or carriage return character. If one knew that neither of those patterns would be encountered, then I believe the above command would work well. Alternatively, both of those problems could be resolved by tokenizing text string values before running the above command, then detokenizing the result.)

Last edited by bmose (2017-09-11 05:11:09 am)

Offline

 

#7 2017-09-10 06:59:32 pm

Shane Stanley
Member
From:: Australia
Registered: 2002-12-07
Posts: 5175

Re: Parsing JSON files

bmose wrote:

Are there any flaws with this straightforward decoding technique?



If there aren't, it's a rare beast. You might care to read this: http://seriot.ch/parsing_json.php

Stuff like parsing JSON and XML is consigned to code libraries for good reason. It can be fun trying to write you're own, but all you're doing is writing bugs waiting to happen.

If there were no alternative, I'd say go for it. But we have several, so IMO this is a really bad idea.


Shane Stanley <sstanley@myriad-com.com.au>
www.macosxautomation.com/applescript/apps/

Offline

 

#8 2017-09-10 07:52:30 pm

bmose
Member
From:: Massachusetts
Registered: 2006-01-03
Posts: 219

Re: Parsing JSON files

Shane Stanley wrote:

You might care to read this: http://seriot.ch/parsing_json.php

I did.  A big problem seems to be the fuzzy nature of the JSON standard and the resulting large number of edge cases that can break a parser.  I've taken to heart your recommendation to stick with code libraries (excepting perhaps where you know your JSON source well and are confident that it will remain simple and well-behaved.)

Last edited by bmose (2017-09-10 07:55:03 pm)

Offline

 

#9 2017-09-10 09:38:07 pm

Shane Stanley
Member
From:: Australia
Registered: 2002-12-07
Posts: 5175

Re: Parsing JSON files

bmose wrote:

excepting perhaps where you know your JSON source well and are confident that it will remain simple and well-behaved.



I'm still not sure why you would bother. I mean, the combination of run script and do shell script isn't exactly fast -- a simple test here shows it takes > 100 times as long as using NSJSONSerialization.


Shane Stanley <sstanley@myriad-com.com.au>
www.macosxautomation.com/applescript/apps/

Offline

 

#10 2017-09-10 10:06:06 pm

bmose
Member
From:: Massachusetts
Registered: 2006-01-03
Posts: 219

Re: Parsing JSON files

You are of course right.  But I guess I still get a thrill out of one-line solutions...sometimes they're just plain fun.

Offline

 

#11 2017-09-10 11:02:53 pm

Shane Stanley
Member
From:: Australia
Registered: 2002-12-07
Posts: 5175

Re: Parsing JSON files

bmose wrote:

But I guess I still get a thrill out of one-line solutions...sometimes they're just plain fun.



Don't we all smile

Applescript:

set theResult to (current application's NSJSONSerialization's JSONObjectWithData:((current application's NSString's stringWithString:jsonString)'s dataUsingEncoding:(current application's NSUTF8StringEncoding)) options:0 |error|:(missing value)) as record


Shane Stanley <sstanley@myriad-com.com.au>
www.macosxautomation.com/applescript/apps/

Offline

 

#12 2017-09-11 05:12:50 am

bmose
Member
From:: Massachusetts
Registered: 2006-01-03
Posts: 219

Re: Parsing JSON files

Touché   smile

P.S. My run script...do shell script... solution takes only 30-40, not >100, times as long as your AppleScriptObjC solution here!!  lol

Last edited by bmose (2017-09-11 06:07:38 am)

Offline

 

#13 2017-09-11 05:18:47 am

Frankv
Member
From:: Amsterdam, NL
Registered: 2017-09-11
Posts: 3

Re: Parsing JSON files

I use jq to parse weather information in json format from Weather Underground.

https://stedolan.github.io/jq/

For example:

Applescript:

set MoonPhase to do shell script "cat " & DirectoryPad & "/AstroDatabase.json | /usr/local/bin/jq -r .moon_phase.phaseofMoon"

Last edited by Frankv (2017-09-11 05:22:14 am)

Offline

 

#14 2017-09-11 05:40:07 am

DJ Bazzie Wazzie
Member
From:: the Netherlands
Registered: 2004-10-20
Posts: 2724
Website

Re: Parsing JSON files

bmose wrote:

But I guess I still get a thrill out of one-line solutions...sometimes they're just plain fun.



When using JavaScript syntax you can write an entire program in a single line. I think the biggest thrill is using less as possible resources and the least instructions.

function run(argv){
	// get current application
	var app = Application.currentApplication();
	// load specific osax
	var StandardAddition = (app.includeStandardAdditions = true, app);
	// define string to display
	var str = "Hello World!";
	// show the dialog
	StandardAddition.displayDialog(str);
};

Is the same as:

function run(argv){var app = Application.currentApplication();var StandardAddition = (app.includeStandardAdditions = true, app);var str = "Hello World!";StandardAddition.displayDialog(str);};
Shane Stanley wrote:

I mean, the combination of run script and do shell script isn't exactly fast -- a simple test here shows it takes > 100 times as long as using NSJSONSerialization.



Of course quite irrelevant in most situations. JSON data is downloaded with the required time and only once. It is changed, and evaluated and the the processing starts.

Shane Stanley wrote:

I'm still not sure why you would bother.



When you have an output from a shell command for example.

Last edited by DJ Bazzie Wazzie (2017-09-11 05:48:50 am)

Offline

 

#15 2017-09-11 06:02:29 am

StefanK
Member
From:: St. Gallen, Switzerland
Registered: 2006-10-21
Posts: 11482
Website

Re: Parsing JSON files

DJ Bazzie Wazzie wrote:

I think the biggest thrill is using less as possible resources and the least instructions.



In my opinion it's using less as possible resources and the most efficient code (regardless of the number of instructions or the number of lines).


regards

Stefan

Offline

 

#16 2017-09-11 06:05:19 am

bmose
Member
From:: Massachusetts
Registered: 2006-01-03
Posts: 219

Re: Parsing JSON files

What really got my juices flowing in this particular problem is the remarkable similarity between JSON and Applescript value specifications.  It seemed almost irresistible simply to do a sed...tr... transform on a well-formed JSON string and run script it into an Applescript object.

Besides the hit in execution speed (which may or may not be relevant in a given usage scenario), the key term is well-formed. If not, the run script...do shell script... approach might well fail.

Offline

 

#17 2017-09-11 06:05:35 am

DJ Bazzie Wazzie
Member
From:: the Netherlands
Registered: 2004-10-20
Posts: 2724
Website

Re: Parsing JSON files

StefanK wrote:
DJ Bazzie Wazzie wrote:

I think the biggest thrill is using less as possible resources and the least instructions.



In my opinion it's using less as possible resources and the most efficient code (regardless of the number of instructions or the number of lines).



That is the least instructions smile Of course with AS you don't know what your instructions are because they're bundled in something we know as commands.

Last edited by DJ Bazzie Wazzie (2017-09-11 06:05:55 am)

Offline

 

#18 2017-09-11 07:08:00 am

Nigel Garvey
Moderator
From:: Warwickshire, England
Registered: 2002-11-20
Posts: 4421

Re: Parsing JSON files

bmose wrote:

JSON's use of square brackets to enclose lists (called arrays in JSON) does not require transformation, since square brackets are an acceptable alternative to curly braces for enclosing Applescript lists.


They used to be (and I think still are) technically different in AppleScript, but it probably won't affect the usability of the resulting lists.

… and the tr command strips any linefeed or carriage return characters:

Applescript:

set applescriptValue to run script (do shell script ("echo " & jsonString's quoted form & " | sed -E 's/\"([^\"]+)\"[[:space:]]*:[[:space:]]*/|\\1|:/g' | tr -d '\\n\\r'"))


It could, of course, all be done in the sed code:  smile

Applescript:

set applescriptValue to run script (do shell script ("echo " & jsonString's quoted form & " | sed -En 's/\"([^\"]+)\"[[:space:]]*:/|\\1|:/g; H; $ {g; s/[[:cntrl:]]+//g; p;}'"))

Last edited by Nigel Garvey (2017-09-11 07:19:24 am)


NG

Offline

 

#19 2017-09-11 07:45:24 am

Frankv
Member
From:: Amsterdam, NL
Registered: 2017-09-11
Posts: 3

Re: Parsing JSON files

Shane Stanley wrote:

I mean, the combination of run script and do shell script isn't exactly fast -- a simple test here shows it takes > 100 times as long as using NSJSONSerialization.



Then I better have a look at my weather program again, I use the "do shell script" with the JQ parser quite a lot.

I almost forgot that before the JQ parser I used something else to parse in AppleScript JSON information. This is the program JSON Helper, with this program you can parse JSON into regular AppleScript lists and records.
http://www.mousedown.net/mouseware/JSONHelper.html

I switched to the jq parser, because for me as a AppleScript beginner this was much easier. I changed some of my jq code to get the same information with JSON Helper.

Applescript:


tell application "JSON Helper"
   set AstroDataBase to fetch JSON from "[url]https://api.wunderground.com/api/My_Key/astronomy/q/NL/Amsterdam.json[/url]"
end tell

set AgeOfMoon to ((AgeOfMoon of moon_phase) in AstroDataBase)

The [ url ] and [ /url ] are not part of my code.

This of course is also not what treed needs, but maybe somebody else finds it useful.

Last edited by Frankv (2017-09-11 07:54:19 am)

Offline

 

#20 2017-09-11 07:48:47 am

Shane Stanley
Member
From:: Australia
Registered: 2002-12-07
Posts: 5175

Re: Parsing JSON files

DJ Bazzie Wazzie wrote:
Shane Stanley wrote:

I'm still not sure why you would bother.



When you have an output from a shell command for example.



I think you're missing my point. I don't see how the source of the output makes any difference to the argument that pre-existing libraries are to be preferred over rolling your own code for things like JSON parsing.

It may affect your choice of library -- you might prefer to pipe it to python, for example -- but that's a different issue.


Shane Stanley <sstanley@myriad-com.com.au>
www.macosxautomation.com/applescript/apps/

Offline

 

#21 2017-09-11 07:59:31 am

bmose
Member
From:: Massachusetts
Registered: 2006-01-03
Posts: 219

Re: Parsing JSON files

Nigel Garvey wrote:

set applescriptValue to run script (do shell script ("echo " & jsonString's quoted form & " | sed -En 's/\"([^\"]+)\"[[:space:]]*:/|\\1|:/g; H; $ {g; s/[[:cntrl:]]+//g; p;}'"))

Nice sed construct!

A little off topic, but my sense is that sed has one of the higher ratios of [hidden power]/[real-life usage] among tools in a coder's toolbox.  Fully tapping into that power, though, takes a great deal of study and experience.

Offline

 

#22 2017-09-11 08:06:07 am

Shane Stanley
Member
From:: Australia
Registered: 2002-12-07
Posts: 5175

Re: Parsing JSON files

StefanK wrote:
DJ Bazzie Wazzie wrote:

I think the biggest thrill is using less as possible resources and the least instructions.



In my opinion it's using less as possible resources and the most efficient code (regardless of the number of instructions or the number of lines).



I must be jaded. I get the biggest thrill from using code I know someone better than me has already debugged.


Shane Stanley <sstanley@myriad-com.com.au>
www.macosxautomation.com/applescript/apps/

Offline

 

#23 2017-09-11 09:58:32 am

Yvan Koenig
Member
Registered: 2006-09-14
Posts: 3201

Re: Parsing JSON files

Hello, two questions:

-- is JSON able to encode Unicode characters ?
-- is SED able to apply to Unicode characters ?

Yvan KOENIG running Sierra 10.12.6 in French (VALLAURIS, France) lundi 11 septembre 2017 16:58:13

Offline

 

#24 2017-09-11 12:26:23 pm

Nigel Garvey
Moderator
From:: Warwickshire, England
Registered: 2002-11-20
Posts: 4421

Re: Parsing JSON files

Yvan Koenig wrote:

-- is JSON able to encode Unicode characters ?


Hi Yvan.

It seems so. (See the script below.)

-- is SED able to apply to Unicode characters ?


sed on Mac OS can handle Unicode as sequences of bytes, but doesn't recognise Unicode characters per se. For instance, you can't use the command y/Д/x/ because sed sees "Д" and "x" as different numbers of characters. But the script below works:

Applescript:

use AppleScript version "2.4" -- Yosemite (10.10) or later
use framework "Foundation"
use scripting additions

set |⌘| to current application

set unicodeText to "⌘řůД⦿"

set originalRecord to {aString:unicodeText, anArray:{unicodeText}}
set jsonData to |⌘|'s class "NSJSONSerialization"'s dataWithJSONObject:(originalRecord) options:(|⌘|'s NSJSONWritingPrettyPrinted) |error|:(missing value)
set jsonString to (|⌘|'s class "NSString"'s alloc()'s initWithData:(jsonData) encoding:(|⌘|'s NSUTF8StringEncoding)) as text

set reconstitutedRecord to run script (do shell script ("echo " & jsonString's quoted form & " | sed -En 's/\"([^\"]+)\"[[:space:]]*:/|\\1|:/g; H; $ {g; s/[[:cntrl:]]+//g; p;}'"))

{jsonString, reconstitutedRecord}
(* -->
{"{
\"aString\" : \"⌘řůД⦿\",
\"anArray\" : [
\"⌘řůД⦿\"
]
}", {aString:"⌘řůД⦿", anArray:{"⌘řůД⦿"}}}
*)

Last edited by Nigel Garvey (2017-09-11 12:28:44 pm)


NG

Offline

 

#25 2017-09-11 12:42:50 pm

Yvan Koenig
Member
Registered: 2006-09-14
Posts: 3201

Re: Parsing JSON files

Thanks a lot Nigel.

Yvan KOENIG running Sierra 10.12.6 in French (VALLAURIS, France) lundi 11 septembre 2017 19:42:43

Offline

 

Board footer

Powered by FluxBB

RSS (new topics) RSS (active topics)