Friday, October 31, 2014

#1 2013-05-10 02:00:08 pm

kel1
Member
Registered: 2013-01-11
Posts: 1819

Sed Context Grep

Hi,

Still going through this tutorial on sed. Can someone tell me if grep cannot do what this sed script cannot do. I added "drain" to the text and it doesn't work. It's supposed to print the line before and after a found pattern:

Applescript:


set t to quoted form of "The
rain
drain
in
Spain
stays
mainly
in
the
plain."

set cmd to "echo " & t & " | sed -n '
/ain/ !{
x
d
}
/ain/ {
x
p
x
p
n
p
a\\
---
x
}'"

do shell script cmd

--> "The
rain
drain
---
in
Spain
stays
---
stays
mainly
in
---
the
plain."

Shouldn't there be:

rain
drain
in
---

for when "drain is found? It appears to not find "drain".

Editted: think I got it. After it finds "rain", it does the printing, then it exchanges drain back into the buffer. Then it goes back to the beginning and reads the next line bypassing the "drain" line. Need to look at this again.

Editted: yes the n read in the "drain" line already. So how do you make the drain line the next line to be read when it's already read and placed in the hold buffer?

Thanks,

Last edited by kel1 (2013-05-10 03:24:40 pm)


Os 10.10
Mbp

Offline

 

#2 2013-05-10 05:55:30 pm

McUsrII
Member
Registered: 2012-11-20
Posts: 2366
Website

Re: Sed Context Grep

Hello.

It isn't easy to cope with every situation, but I think you can have a pattern within the pattern! smile

By the way, I recently came to learn that the regexp algorithms used in awk, and sed, can be up to a million times faster, than those used in perl, ruby, python and Java! (When the patterns gets compicated enough.

It is good there are some benefits to use Applescript, so we can get some speed somewhere, as the do shell script doesn't count for much, once the files or patterns are big. Here is a link to the article..

By the way, I have secured a utility you can use for writing to a terminal window from a do shell script, a tiny bit, as the commands are executed with the rights of the  holder of the terminal window, so I have only turned off signal handling, while the command is active in doing stuff. There is a link to it here with source. I should mention that DJ Bazzie Wazzie brought it to my notice, and that you can read all about it here

Last edited by McUsrII (2013-05-10 06:03:50 pm)


Filed under: sed

Offline

 

#3 2013-05-10 06:22:32 pm

kel1
Member
Registered: 2013-01-11
Posts: 1819

Re: Sed Context Grep

Hi McUsrII,

I haven't read the article yet, but I know what you're talking about the cost of calling 'do shell script'. Once you do one call then If it's worth the call then it's ok. Depends on the situation. smile

It seems that AppleScript programmers have been doing their job, except for the result window in Script Editor showing results.

So, I want to do as many things as I can, If I do have to 'do shell script'. Only after doing it can I know which is better; AppleScript or 'do shell scirpt'.

In the mean time, I find sed fun for some reason. It kind of reminds me of the olden days with assembly language. I used to have the old commodore with the two processors.

But I'm diversing.

BTW, I timed a 'do shell script' call vs AppleScript and ran it a thousand times. Not calling the unix was like a thousand times faster. The shell script took about 6 seconds and the applescript to half a second. What the overhead of calling the 'do shell script' really depends.

Editted: actually Vanilla AppleScript was 500 times faster because it took .05 secs I think and the shell script took almost 6 seconds.

Last edited by kel1 (2013-05-10 06:55:55 pm)


Os 10.10
Mbp

Offline

 

#4 2013-05-10 07:46:21 pm

McUsrII
Member
Registered: 2012-11-20
Posts: 2366
Website

Re: Sed Context Grep

Hello.

I have actually gotten hold some fast 30 ines of c, so I could have had implemented it in Applescript, but I provide the ink here should anybody feel indulged…

I use sed, because it is fast, it is not bloated like perl, but has really a small command set, that must be practiced now and then. smile It is really simple, just a loop, a pattern space, and a holdspcace. But the terseness, and the abstraction level combined, makes it a little bit tricky. I never fiddle with it in Applescript before I have gotten it right in a Terminal, because the edit-test cycle is shorter there, even if it is in a sed script.


Filed under: sed

Offline

 

#5 2013-05-10 08:11:28 pm

kel1
Member
Registered: 2013-01-11
Posts: 1819

Re: Sed Context Grep

McUsr, your grasp of what's going on is amazing. Tricky is good and it makes it fun. There are many ways to do things.

I never liked c in school. I liked Pascal better back then. The wording was more like a higher level language. Maybe that's why I never became a programmer.

I don't know why I started reminiscing, but one day I was on the internet back in about1981. Trying to connect to a game. When I first saw typing on my screen, it was great. The site was giving directions, all text. big_smile Compared to now, it's unreal man.

Editted: no it was about 1984.

Last edited by kel1 (2013-05-10 08:13:33 pm)


Os 10.10
Mbp

Offline

 

#6 2013-05-11 01:37:04 am

DJ Bazzie Wazzie
Member
From: the Netherlands
Registered: 2004-10-20
Posts: 1969

Re: Sed Context Grep

kel1 wrote:

I never liked c in school. I liked Pascal better back then. The wording was more like a higher level language. Maybe that's why I never became a programmer.

Are you sure about that? Pascal is the same level as C and the performance depends on the compiler. Why Pascal is fading away is simply because Apple stopped developing their OS at the early '90s in Pascal which makes it no longer an OS language. Pascal has proven itself, in the time that every bit and cycle counts, to be good as C.


Kind regards

Offline

 

#7 2013-05-11 04:25:50 am

McUsrII
Member
Registered: 2012-11-20
Posts: 2366
Website

Re: Sed Context Grep

Hello.


There is a matter of fact a good free Pascal compiler out there, I haven't had the time to check it out, GNU Pascal, that supports almost every pascal dialect there were. I'd look for that if you want to have a go with it.

The problem with pascal as I see it, is the lack of standardized libraries, and that it is a kind of theoretical language. It is nicer than C, but the "clean" approach makes it, (or made it when I used it), much more limiting.

Yesterday, I wrote a c-program, that gotoe'd into the middle of a case statement, and installed a longjumphandler, gotoed out again, the longjump  was later being used from within a signal, that were issued, when the program was awakened again, after it had suspended itself.  It worked! big_smile Try that in Pascal! (You can probably do that but…).

Last edited by McUsrII (2013-05-11 04:27:26 am)


Filed under: Pascal

Offline

 

#8 2013-05-11 06:21:45 am

DJ Bazzie Wazzie
Member
From: the Netherlands
Registered: 2004-10-20
Posts: 1969

Re: Sed Context Grep

It is possible within Pascal but like in many Programming language (including C) a goto is considered as lack of design and bad software development and should be the programmer's last choice. So if goto in Pascal is easier or worse than in C, it still is considered as equally bad/poor designed. However some programming language don't have a block design like C or Pascal and uses goto (on line number) instead like in BBC Basic.

Last edited by DJ Bazzie Wazzie (2013-05-11 06:22:01 am)


Kind regards

Offline

 

#9 2013-05-11 06:25:43 am

McUsrII
Member
Registered: 2012-11-20
Posts: 2366
Website

Re: Sed Context Grep

Hello.

Sometimes a goto is the solution, sometimes a longjump, but they should be avoided if possible, and used sparsely, they are ok, when they simplify things in total, and I guess that is why it is included. It is not to be used like in BBC basic. smile But only when there aren't any good structured solution.

Offline

 

#10 2013-05-11 06:35:13 am

Nigel Garvey
Moderator
From: Warwickshire, England
Registered: 2002-11-19
Posts: 3553

Re: Sed Context Grep

Hi.

I'm not sure this is the best way to do it, but it seems to work with the given text and variations. Not quite the same as grep -C though. I've left it uncommented.  wink

Applescript:

set t to quoted form of "The
rain
drain
in
Spain
stays
mainly
in
the
plain"

set cmd to "echo " & t & " | sed -n '
1 h
2,$ {
   /ain/ !{
       x
       /ain/ {
           G
           p
           $ !s/^.*$/---/p
           c\\'$'\\n''
       }
   }
   /ain/ {
       x
       /ain/ {
           G
           p
           s/^[^[:cntrl:]]*\\n//
           h
           $ !s/^.*$/---/p
           $ c\\'$'\\n''
       }
       /---/ !{
           s/^[^[:cntrl:]]*\\n//
           G
           h
       }
   }
}
$ {
   g
   /ain/ p
}'"

do shell script cmd


NG

Offline

 

#11 2013-05-11 07:25:10 am

McUsrII
Member
Registered: 2012-11-20
Posts: 2366
Website

Re: Sed Context Grep

cool

Offline

 

#12 2013-05-12 02:18:02 am

kel1
Member
Registered: 2013-01-11
Posts: 1819

Re: Sed Context Grep

I finally got how to make it loop at the right place in the script, so it could print the lines before and after the pattern(s). Not sure, if this is how grep works when there are consecutive matching lines. I need to check out how Nigel's results look.

Applescript:


set t to quoted form of "The
rain
train
in
Spain
stays
mainly
in
the
stain
plain."

set cmd to "echo " & t & " | sed -n '
/ain/ !{
x
d
}
/ain/ {
x
# print the line before
p
# get a copy of the matching lline
x
# print the matching line
p
:loop
# clear pattern space and get next line
n
# print next line
p
# branch if line contains pattern
/ain/ b loop
# print divider
a\\
---
x
}'"

do shell script cmd

Thanks a lot,


Os 10.10
Mbp

Offline

 

#13 2013-05-12 03:19:31 am

Nigel Garvey
Moderator
From: Warwickshire, England
Registered: 2002-11-19
Posts: 3553

Re: Sed Context Grep

Your results are closer to grep's than mine. It merges overlapping contexts.  cool

Applescript:

set t to quoted form of "The
rain
train
in
Spain
stays
mainly
in
the
stain
plain."

set cmd to "echo " & t & " | grep -C1 'ain'"
do shell script cmd

Last edited by Nigel Garvey (2013-05-12 03:32:49 am)


NG

Offline

 

#14 2013-05-12 04:28:51 am

McUsrII
Member
Registered: 2012-11-20
Posts: 2366
Website

Re: Sed Context Grep

Hello.

It is very admirable kel! smile

Here is a technique for using sed, I have been playing with, in order to make it more versatile, (or to use as little as possible of it.)

The idea here, is to use sed in combination with other tools, where you really have as much as you can of precomposed input in separate files, and leverages upon that you can specifiy stdin as either "-" /dev/fd/0 or /def/stdin, (one of these normallly works.).

Ok, so I have a preamble of html, called file1, containing up to and including the body tag, and I have a "postamble", containting the closing body tag and the rest of it, and a list, of say greek letters spelled out in latin, and the sed script below to make an unordered list:

Code:

#!/usr/bin/sed -nf
1i\
<ul>\

s_^.*_ <li>&</li>_p
$a\
</ul>\

Then, in order to generate an html file with the result, I may use a commandline like this:

Code:

cat alfabet |./ul.sed |cat file1 - file2 >alfa.html

Last edited by McUsrII (2013-05-12 04:29:54 am)


Filed under: sed

Offline

 

#15 2013-05-12 01:36:56 pm

DJ Bazzie Wazzie
Member
From: the Netherlands
Registered: 2004-10-20
Posts: 1969

Re: Sed Context Grep

In AWK it can be like with overlap:

Applescript:


do shell script "awk 'BEGIN{first=1}/ain/{
   if (first==0)
       print \"---\"
   print head
   print $0
   getline
   print $0
   first=0 };{head=$0}' <<< "
& t

or without overlap:

Applescript:


do shell script "awk 'BEGIN{first=1}/ain/{
   if (first==0)
       print \"---\"
   print head
   print $0
   getline
   print $0
   getline
   first=0 };{head=$0}' <<< "
& t


Kind regards

Offline

 

#16 2013-05-12 03:02:28 pm

McUsrII
Member
Registered: 2012-11-20
Posts: 2366
Website

Re: Sed Context Grep

One day soon, I'll write one in C that mimick's Nigel Garvey's version in sed, but with filenames and line numbers.

cgrep is a tool I hopefully don't use to much, but it is really handy when you are tracking a variable or something, as useful as diff, for detecting changes between two versions…

It feels kind of weird for starters, to just look at fragments of files, but I experienced that as a boon once I got used to it: I don't have to navigate, and I only look at the parts that interests me, there and then.

Last edited by McUsrII (2013-05-12 03:07:27 pm)


Filed under: sed

Offline

 

#17 2013-05-12 03:22:26 pm

kel1
Member
Registered: 2013-01-11
Posts: 1819

Re: Sed Context Grep

Hi,

Nigel,

Your script was actually what I was thinking it should be with the text as is. When I add a matching pattern in the first line:

Applescript:


set t to quoted form of "The brain
rain
drain
in
Spain
stays
mainly
in
the
plain"

set cmd to "echo " & t & " | sed -n '
1 h
2,$ {
   /ain/ !{
       x
       /ain/ {
           G
           p
           $ !s/^.*$/---/p
           c\\'$'\\n''
       }
   }
   /ain/ {
       x
       /ain/ {
           G
           p
           s/^[^[:cntrl:]]*\\n//
           h
           $ !s/^.*$/---/p
           $ c\\'$'\\n''
       }
       /---/ !{
           s/^[^[:cntrl:]]*\\n//
           G
           h
       }
   }
}
$ {
   g
   /ain/ p
}'"

do shell script cmd

The second and third matches don't come out right.

Hi DJ Bazzie Wazzie,

I have a new library; awk. Thought I'd start with sed first because I've tried to learn it before, but the tutorials on awk seemed complicated. Thanks for the intro.

Well, my biorythm's intellectual line is going up (just passed zero), so now is the time to learn this stuff. smile


Os 10.10
Mbp

Offline

 

#18 2013-05-12 04:11:55 pm

kel1
Member
Registered: 2013-01-11
Posts: 1819

Re: Sed Context Grep

Just remembered what I read on the internet somewhere, that grep cannot print the same lines more then once. That's why the grep output is incomplete and maybe has those double dividers sometimes.

Editted: I don't know what the person meant by this. Grep seems to print the same line. Need to read that post again. if I can find it. I probably read it wrong.

Last edited by kel1 (2013-05-12 05:31:14 pm)


Os 10.10
Mbp

Offline

 

#19 2013-05-12 04:21:09 pm

Nigel Garvey
Moderator
From: Warwickshire, England
Registered: 2002-11-19
Posts: 3553

Re: Sed Context Grep

kel1 wrote:

When I add a matching pattern in the first line … The second and third matches don't come out right.

I could have sworn I tested with that before I posted. hmm  Obviously not. Sorry. I'll try and fix it — but not tonight.


NG

Offline

 

#20 2013-05-13 07:15:46 am

DJ Bazzie Wazzie
Member
From: the Netherlands
Registered: 2004-10-20
Posts: 1969

Re: Sed Context Grep

kel1 wrote:

Hi DJ Bazzie Wazzie,

I have a new library; awk. Thought I'd start with sed first because I've tried to learn it before, but the tutorials on awk seemed complicated. Thanks for the intro.

You're welcome. cool

It's good to start with regular expressions first and then start with AWK. AWK, the predecessor of Perl, is much more versatile which can lead you to a leap in the dark. When you're already used to C-style syntaxes and write in a more decompressed style, AWK scripts are easier to read than sed IMO. The latest version of Kernighan (who also developed C) is considered as the one and only true AWK version which is standard on FreeBSD (also Mac OS X). I would also recommend to book associated to this version AWK, ISBN 0-201-07981-X (which can be found at the bottom in the AWK man page), which tells you everything about AWK what you should know about it.


Kind regards

Offline

 

#21 2013-05-13 07:37:10 am

Nigel Garvey
Moderator
From: Warwickshire, England
Registered: 2002-11-19
Posts: 3553

Re: Sed Context Grep

OK. A slight rethink wherein the hold space acts similarly to a three-line FIFO stack, except that the lines are all retrieved at once and only the last two are put back before another is added. The stack is output complete if its penultimate line contains "ain", with adjustments at the last line. It seems to work.

Applescript:

set t to quoted form of "The brain
rain
drain
in
Spain
stays
mainly
in
the
Staines
plain
again
Jane

again"


set cmd to "echo " & t & " | sed -En '
# Move the first line to the “stack” (hold space)
1 h
# With each of the following lines in turn:
1 !{
   # Append the line to the stack and retrieve the stack contents.
   H
   g
   # If the penultimate retrieved line contains “ain”, print them all and, if not at the last line of text, output a separator.
   /ain[^[:cntrl:]]*\\n[^[:cntrl:]]*$/ {
       p
       $ !i\\'$'\\n''---
   }
   # If three lines were taken from the stack, lose the first one and push the other two back.
   /^[^[:cntrl:]]*\\n([^[:cntrl:]]*\\n[^[:cntrl:]]*)$/ {
       s//\\1/
       h
   }
}
# At the last line of the text, after the above processes, the pattern space contains either the first line (if there’s only one) or the last two (the edited stack contents).
$ {
   # If the line contains “ain”:
   /ain[^[:cntrl:]]*$/ {
       # If there’s a line before it also containing “ain”, output another separator.
       /ain[^[:cntrl:]]*\\n/ i\\'$'\\n''---
       # Print the pattern space contents.
       p
   }
}'"


do shell script cmd

Edit: Comments revised.

Last edited by Nigel Garvey (2013-05-14 03:15:28 am)


NG

Offline

 

#22 2013-05-13 08:19:18 am

McUsrII
Member
Registered: 2012-11-20
Posts: 2366
Website

Re: Sed Context Grep

sad The lines aren't numbered. lol

Just kidding. It's just Brilliant!cool


Filed under: sed

Offline

 

#23 2013-05-13 09:55:06 am

McUsrII
Member
Registered: 2012-11-20
Posts: 2366
Website

Re: Sed Context Grep

Actually, I figured, I could write what I wanted faster in sed, and implement it in a shell script, rather than write it in c. (The sole intent is to have a tool to trace a variable/function call throughout a source code file.)

What I want, is a short context, and that the context is to be broken, when a new match appear, and I want line numbers.

Given the latest "poem" from the post above, the output looks like this:

Code:

1 The brain
---
2 rain
---
3 drain
4 in
---
5 Spain
6 stays
---
7 mainly
8 in
---
9 the
10 Staines
---
11 plain
---
12 again
13 Jane
---
14
15 again"

I implemented it as a shell script, which should be easy to use from a do shell script, the regexp is given as a parameter quoted with double quotes on the commandline, and the input must be redirected into it.

Code:

#!/bin/bash
if [ $# -ne 1 ] ; then
echo "Usage: cgrep \"pattern\" <input

Prints 1 line before and after a match,
and adds three dashes to split the
sequences. Linenumbers for a match is
added.
Cgrep breaks the context if the next
line is a match by itself.
" >/dev/tty
exit 2
fi
nl -b a |/usr/bin/sed -n '
1 {
/'"$1"'n/! h
/'"$1"'/ {
p
a\
---\

n
}
}
:op
1! {
/'"$1"'/! {
h
n
}
/'"$1"'/ {
x
/'"$1"'/ {
g
p
}
/'"$1"'/! {
p
g
p
}
n
/'"$1"'/ {
i\
---\

b op
}
/'"$1"'/! {
p
a\
---\

}
}
}
'

Last edited by McUsrII (2013-05-13 10:22:44 am)


Filed under: sed

Offline

 

#24 2013-05-13 03:40:08 pm

McUsrII
Member
Registered: 2012-11-20
Posts: 2366
Website

Re: Sed Context Grep

Some stuff I made from ideas acquired  in this thread! smile

Here is one little tool to wrap something into html, by piping stuff into it. (Its' called html_wrap.)

Code:

#!/bin/bash
echo '<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"'
echo ' "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">'
echo '<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">'
echo '<head>'
echo ' <meta http-equiv="content-type" content="text/html; charset=utf-8" />'
echo ' <title>Wrapper</title>'
echo '</head>'
echo '<body>'
cat -
echo '</body>'
echo '</html>'

Below is a script that shows your shell scripts and the like with qlmanager (quicklook.). I have called it "g"

Code:

#!/bin/bash
if [ $# -ne 1 ] ; then
echo "Usage: g file.
shows a file probably script with quicklook." >/dev/tty
exit 2
fi
if [ ! -e $1 ] ; then
echo "g: Can't find $1" >/dev/tty
exit 1
fi
a=$(basename $1)
(
echo '<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"'
echo ' "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">'
echo '<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">'
echo '<head>'
echo ' <meta http-equiv="content-type" content="text/html; charset=utf-8" />'
echo ' <title>Wrapper</title>'
echo '</head>'
echo '<body>'
echo '<pre><code>'
cat $1 |sed -e 's/</\&lt;/g' -e 's/>/\&gt;/g'
echo '</code></pre>'
echo '</body>'
echo '</html>'
) >/tmp/$a.html
qlmanage -px /tmp/$a.html 2>&1 >/dev/null &

You need to chmod u+x to use both of course, and store them somwhere in your bin.

The idea is of course to use "g" from a do shell script, so that you can look at the scripts from Finder. smile

Edit

I have made a minor change, removing the path when generating the html file, so it works properly outside the folder it is placed. smile

Here is a small applescript, that calls it with the first item of the current Finder selection as argument.

Applescript:

tell application "Finder"
   set a to its selection as alias list
   if a is not missing value then
       if ((count a) > 1) then
           set a to first item of a
           set a to POSIX path of a
       else
           set a to POSIX path of a
       end if
       do shell script "g " & quoted form of a & " 2>&1 >/dev/null &"
   end if
end tell

By the way, I saved the applescript above as an applet, found an icon for it, and dragged it onto the toolbar, just great! big_smile

Last edited by McUsrII (2013-05-14 05:35:18 am)


Filed under: g

Offline

 

#25 2013-05-13 11:19:41 pm

kel1
Member
Registered: 2013-01-11
Posts: 1819

Re: Sed Context Grep

Wow, everybody's been busy! Yeah, nice poem. big_smile

I tried to do this script with just the lower case commands for now. Finally got the loop in the right place. Had to hold the matching line in the buffer, incase the next line had a match also. Had to use insert (i) instead of append, because append only appends after all lines are printed within the curly braces.

Applescript:


set t to quoted form of "The again
rain
train
in
Spain
stays
mainly
in
the
brain
stain
plain."

set cmd to "echo " & t & " | sed -n '
/ain/ !{
x
d
}
/ain/ {
:loop
x
# print the line before
p
# get a copy of the matching lline
x
# print the matching line
p
# hold this lines in case next line contains pattern
h
# clear pattern space and get next line
n
# print next line
p
# branch if line contains pattern
# insert divider here
i\\
---
/ain/ b loop
x
}'"

do shell script cmd

This method is limited though, to one line before and after a match. I'll have to learn Nigel's method in the following sections with the capitalized commands for more veratillity.

McUsr,

How did you know I was planning on learning html? You must be a psychic. smile Actually, I don't really know what is going on in your scripts yet. I've been walking around in a daze, thinking about where to put the loop. Now I can reread the posts.

Editted: oops, posted the wrong script.

Applescript:


set t to quoted form of "The brain
rain
drain
in
Spain
stays
mainly
in
the
Staines
plain
again
Jane

again"

set cmd to "echo " & t & " | sed -n '
/ain/ !{
x
d
}
/ain/ {
:loop
x
# print the line before
p
# get a copy of the matching lline
g
# print the matching line
p
# clear pattern space and get next line
n
# print next line
p
# insert divider here
i\\
---
# branch if line contains pattern
/ain/ b loop
x
}'"

do shell script cmd

Thanks a lot everybody,
It's been fun.

Last edited by kel1 (2013-05-14 12:30:34 pm)


Os 10.10
Mbp

Offline

 

Board footer

Powered by FluxBB

[ Generated in 0.046 seconds, 10 queries executed ]

RSS (new topics) RSS (active topics)