Stripping non printing characters

[gregjsmith]

I’ve got a lot of duplicate Address Book entries with what I can best describe as non printing characters in the name. I’m trying to build a routine to strip them out. When I ask Applescript what they are it shows them as “” or ASCII 0. I’m not sure what I need to strip out.

[cwtnospam]

This should get you started:

set x to quoted form of "Sample text. smdm#$%%&"
set y to do shell script "echo " & x & " | perl -pe 's/[^[:alnum:]|[:space:]]//g'"

From man grep:

A list of characters enclosed by [ and ] matches any single character
in that list; if the first character of the list is the caret ^ then it
matches any character not in the list. For example, the regular
expression [0123456789] matches any single digit. A range of charac-
ters may be specified by giving the first and last characters, sepa-
rated by a hyphen. Finally, certain named classes of characters are
predefined. Their names are self explanatory, and they are [:alnum:],
[:alpha:], [:cntrl:], [:digit:], [:graph:], [:lower:], [:print:],
[:punct:], [:space:], [:upper:], and [:xdigit:]. For example,
[[:alnum:]] means [0-9A-Za-z], except the latter form depends upon the
POSIX locale and the ASCII character encoding, whereas the former is
independent of locale and character set. (Note that the brackets in
these class names are part of the symbolic names, and must be included
in addition to the brackets delimiting the bracket list.) Most
metacharacters lose their special meaning inside lists. To include a
literal ] place it first in the list. Similarly, to include a literal
^ place it anywhere but first. Finally, to include a literal - place
it last.

   The period .  matches any single character.  The symbol \w is a synonym
   for [[:alnum:]] and \W is a synonym for [^[:alnum]].

   The caret ^ and the dollar sign $ are metacharacters that  respectively
   match the empty string at the beginning and end of a line.  The symbols
   \< and \> respectively match the empty string at the beginning and  end
   of  a  word.   The  symbol \b matches the empty string at the edge of a
   word, and \B matches the empty string provided it's not at the edge  of
   a word.

[regulus6633]

I had this problem recently. The cause of my problem was the file encoding was utf-16. There’s no way I could find to change the file encoding. Instead I read the file into applescript, used text item delimiters to remove the offending character, then you can work with the text or write it back out to a new file.

To use text item delimiters I first needed the character I wanted to remove. So I just copied it from the results section in script editor. Then you can paste the copied character into this script and use it to remove the characters…

==assuming the varible badText has the text with the unwanted character==

set applescript’s text item delimiters to “” → put the copied character between the quotes!
set tidText to text items of badText
set applescript’s text item delimiters to “” → this time nothing goes between the quotes
set goodText to tidText as unicode text