Using sed to search and replace literal text?

snaplash · February 26, 2017, 12:21am

Hi All,

It has been a long and arduous journey to create my app. I started in Applescript as I was very new to coding and it was wonderful to be able to make my own mistakes…As I continued, most of my code ended up being translated to shell.

Since then, I am pretty much bash based held together via AS

I was wondering if anyone has any input on using a terminal based solution via homebrew to replace literal text.

Right now I am automating textwrangler (which works flawlessly) as I can replace text literally, but I am looking for a headless solution that doesnt open up a GUI program.

eg:

tell application “TextWrangler”
open “/Users/snaplash/Desktop/SNAPLASH_DO_NOT_DELETE/PROCESSING_DO_NOT_DELETE/” & listingMLSProviderPrefix of theRecord & Mlsid of theRecord & “_MARKER_20.html” as POSIX file
tell front document
set tid to AppleScript’s text item delimiters
set AppleScript’s text item delimiters to “/”
set line 85 to “var address = '” & listingStreetNumber of theRecord & ", " & listingCity of theRecord & " " & listingState of theRecord & " " & listingZipcode of theRecord & “';”
set line 122 to "icon: " & quote & “file://” & “/Users/snaplash/Desktop/SNAPLASH_DO_NOT_DELETE/CONSTANTS_DO_NOT_DELETE/LOGO_VAULT_DO_NOT_DELETE/” & brandingFolderName of theRecord & “/” & brandingFolderName of theRecord & “_EARTH_MARKER.png” & quote & “,”
end tell
save front document
close front document
quit
end tell

I am running into many issues with sed due to regex special characters.

I have seen vi or vim? Can anyone provide feedback on a text editor via terminal that will work and will be transportable to a server based system once I complete the beta of my program?

Thank you all in advance!

Mark

Nigel_Garvey · February 26, 2017, 7:37pm

Hi Mark.

Well. The replacement of entire numbered lines is basically as simple as can be with sed:

set filePath to "/Users/snaplash/Desktop/SNAPLASH_DO_NOT_DELETE/PROCESSING_DO_NOT_DELETE/" & listingMLSProviderPrefix of theRecord & Mlsid of theRecord & "_MARKER_20.html"
set replacementLine85 to "var address = '" & listingStreetNumber of theRecord & ", " & listingCity of theRecord & " " & listingState of theRecord & " " & listingZipcode of theRecord & "';"
set replacementLine122 to "icon: " & quote & "file://" & "/Users/snaplash/Desktop/SNAPLASH_DO_NOT_DELETE/CONSTANTS_DO_NOT_DELETE/LOGO_VAULT_DO_NOT_DELETE/" & brandingFolderName of theRecord & "/" & brandingFolderName of theRecord & "_EARTH_MARKER.png" & quote & ","

do shell script "sed '85 s|^.*$|" & replacementLine85 & "|
122 s|^.*$|" & replacementLine122 & "|' " & quoted form of filePath

The flies in the ointment when trying to replicate your TextWrangler script with sed are the possibilty of reserved sed characters in the replacement strings and the fact that writing back to the original file with sed leaves you with an empty file.

I’ve used vertical bars for the ‘s’ command delimiters above as there are several forward slashes in one of the replacement strings. If the strings contain ampersands, single-quotes (apostrophes), or backslashes, these need to be escaped. I hope there are no backslashes as they’ve totally defeated me for the moment.

There are more convenient ways than sed to do this, but it’s Sunday.

set filePath to "/Users/snaplash/Desktop/SNAPLASH_DO_NOT_DELETE/PROCESSING_DO_NOT_DELETE/" & listingMLSProviderPrefix of theRecord & Mlsid of theRecord & "_MARKER_20.html"
set replacementLine85 to "var address = '" & listingStreetNumber of theRecord & ", " & listingCity of theRecord & " " & listingState of theRecord & " " & listingZipcode of theRecord & "';"
set replacementLine122 to "icon: " & quote & "file://" & "/Users/snaplash/Desktop/SNAPLASH_DO_NOT_DELETE/CONSTANTS_DO_NOT_DELETE/LOGO_VAULT_DO_NOT_DELETE/" & brandingFolderName of theRecord & "/" & brandingFolderName of theRecord & "_EARTH_MARKER.png" & quote & ","

do shell script "# Join the two replacement lines with a linefeed, pass the quoted form of the result to sed to escape the linefeed and any ampersands or bars, and assign the result to a variable.
doctoredReplacements=$(sed -E '
# Insert a backslash at the end of the first replacement line to escape the linefeed and insert backslashes before any ampersands or bars in either line.
1 s/$/\\\\/
s/&|\\|/\\\\&/g' <<<" & quoted form of (replacementLine85 & linefeed & replacementLine122) & " );

# Read the file at filePath into sed to replace lines 85 and 122 with the doctored replacement lines. Assign the edited text to a variable.
editedText=$(sed '
# Replace line 85 with the block of two replacement lines, then delete the second.
85 {
	s|^.*$|'\"$doctoredReplacements\"'|
	s|\\n.*$||
}
# Replace line 122 with the block of two replacement lines, then delete the first.
122 {
	s|^.*$|'\"$doctoredReplacements\"'|
	s|^.*\\n||
}' " & quoted form of filePath & ") ;

# Retrieve the edited text from the variable and write the result back to the original file.
echo \"$editedText\" >" & quoted form of filePath

Edit: For comparison, the fast, simple, vanilla method would be:

set filePath to "/Users/snaplash/Desktop/SNAPLASH_DO_NOT_DELETE/PROCESSING_DO_NOT_DELETE/" & listingMLSProviderPrefix of theRecord & Mlsid of theRecord & "_MARKER_20.html"
set replacementLine85 to "var address = '" & listingStreetNumber of theRecord & ", " & listingCity of theRecord & " " & listingState of theRecord & " " & listingZipcode of theRecord & "';"
set replacementLine122 to "icon: " & quote & "file://" & "/Users/snaplash/Desktop/SNAPLASH_DO_NOT_DELETE/CONSTANTS_DO_NOT_DELETE/LOGO_VAULT_DO_NOT_DELETE/" & brandingFolderName of theRecord & "/" & brandingFolderName of theRecord & "_EARTH_MARKER.png" & quote & ","

set astid to AppleScript's text item delimiters
set fileAccess to (open for access POSIX file filePath with write permission)
try
	set textLines to paragraphs of (read fileAccess as «class utf8»)
	set item 85 of textLines to replacementLine85
	set item 122 of textLines to replacementLine122
	set AppleScript's text item delimiters to linefeed
	set editedText to textLines as text
	set AppleScript's text item delimiters to astid
	
	set eof fileAccess to 0
	write editedText to fileAccess as «class utf8»
	close access fileAccess
on error errMsg
	close access fileAccess
	set AppleScript's text item delimiters to astid
	display dialog errMsg buttons {"OK"} default button 1
end try

snaplash · February 27, 2017, 11:52pm

Nigel,

Thank you! You have saved me again!

sed was obviously the way to go but sometimes I just cant seem to get things to work.

As always, you have come through!

I cannot tell you how much you have helped me!

Regards,

Mark

Nigel_Garvey · February 28, 2017, 12:46pm

Hi Mark.

Thanks for the feedback. I’ve just found out how to get sed’s “edit files in-place” option to work, which allows slightly more compact code:

set filePath to "/Users/snaplash/Desktop/SNAPLASH_DO_NOT_DELETE/PROCESSING_DO_NOT_DELETE/" & listingMLSProviderPrefix of theRecord & Mlsid of theRecord & "_MARKER_20.html"
set replacementLine85 to "var address = '" & listingStreetNumber of theRecord & ", " & listingCity of theRecord & " " & listingState of theRecord & " " & listingZipcode of theRecord & "';"
set replacementLine122 to "icon: " & quote & "file://" & "/Users/snaplash/Desktop/SNAPLASH_DO_NOT_DELETE/CONSTANTS_DO_NOT_DELETE/LOGO_VAULT_DO_NOT_DELETE/" & brandingFolderName of theRecord & "/" & brandingFolderName of theRecord & "_EARTH_MARKER.png" & quote & ","

do shell script "# Join the two replacement lines with a linefeed, pass the quoted form of the result to sed to escape the linefeed and any ampersands or bars, and assign the result to a variable.
doctoredReplacements=$(sed -E '
# Insert a backslash at the end of the first replacement line to escape the linefeed and insert backslashes before any ampersands or bars in either line.
1 s/$/\\\\/
s/&|\\|/\\\\&/g' <<<" & quoted form of (replacementLine85 & linefeed & replacementLine122) & " );

# Edit the file in-place using sed.
sed -i '' '
# Replace line 85 with the block of two replacement lines, then delete the second line from the replacement.
85 {
    s|^.*$|'\"$doctoredReplacements\"'|
    s|\\n.*$||
}
# Replace line 122 with the block of two replacement lines, then delete the first line from the replacement.
122 {
    s|^.*$|'\"$doctoredReplacements\"'|
    s|^.*\\n||
}' " & quoted form of filePath

Notes and caveats:

Editing files in-place is achieved with sed’s -i option.
On my 10.11 system, this option only works if it’s the first one specified, contrary to the impression given by the synopsis in the man document. (Correction: It turns out that the -i option can go after the command string as long as the command string itself is preceded by -e. ie. "sed -e ‘blah blah blah’ -i ‘’ " & quoted form of filePath)
Using -i ‘’ as above causes the original file to be edited in-place. The man document doesn’t recommend this “as you risk corruption or partial content in situations where disk space is exhausted, etc.”
Using -i ‘.extn’ causes the original file simply to have “.extn” appended to its name and a new file containing the edited text to be created with the original name in the same folder. This new file may not become visible on screen until something is done to force the Finder to update its display, such as changing the View of the relevant window or running a script telling the Finder to ‘update’ or ‘reveal’ the file.
Apart from that, it’s all quite straightforward.