Remove comments from file while ignoring quoted comment signs

Question

I would like to remove comments starting with # from a file. I have tried the simpler approaches described in How can I remove all comments from a file? but I have a few additional rules:

• A # does not start a comment if it occurs as part of a quoted string.
• Strings can be quoted by single quotes ' or double quotes ".
• Double-quoted strings can contain quotes if preceded by a backslash \", backslashes are quoted as \\.
• All quotes in the input are matched. However, this is not required for quotes that are part of a string's content, in other words "'", "\"" and '"' are valid strings).
• Quoted strings can't contain newline characters.
• Comments can contain any characters including any number of #, ', " and \.
• Any # outside of quotes starts a comment (as Stéphane Chazelas pointed out code code for most shells follows more complex rules - think about Bash's $# which does not start a comment).

For example the following input

# comment only
# comments are allowed to contain quotes "' and # number signs
# comments are allowed to contain pairs 'of' "quotes"
some text # with an explanation
some "quoted text # not a comment" # comment
'# not a comment' and '# not a comment either' # comment
"# not a comment containing 'quotes\"" # another comment

shall be converted into the following output

some text
some "quoted text # not a comment"
'# not a comment' and '# not a comment either'
"# not a comment containing 'quotes""

I would like to accomplish this with popular Unix command line tools like awk, grep and sed on modern Debian/Ubuntu systems. I'm not strictly limited to features described by POSIX although a POSIX-compliant solution would be preferred.

Answer 1

If the point is to remove comments from POSIX sh scripts, note that only the ones marked as YES in the code below are comments:

echo 1 # YES
echo 2 $# NO foo# NO
echo 3;#YES
# YES
cat << E
# NO
E
echo 4 " # NO \" # NO" \" # YES
echo "5
# NO
$(echo 6 # YES
)
`echo 7 \" # NO \"`
"
eval 'echo 8 # NO, then YES'

(and you can see the stackexchange syntax highlighter gets it wrong in most of the cases).

Covering those would take hundreds of lines of awk or sed code.

And rules for csh, fish, perl, python, ruby which are other languages that have "..." and '...' quotes and # as comment leader would be radically different.

If

• it's not about shell syntax,
• you can assume that there is no escaping of quotes,
• that the quoted strings don't contain newline characters,
• that all quotes are matched,
• that any # outside of quotes starts a comment and not only those following a blank or other delimiter,
• that the input is valid text in the current locale

And if by standard you mean POSIX 2018 or earlier, you could do it with sed with:

sed "s/^\(\(\([^\"'#]\)*\(\"[^\"]*\"\)\{0,1\}\('[^']*'\)\{0,1\}\)*\)#.*/\1/"

POSIX 2018 sed doesn't support -E for EREs which would be needed for an alternation operator, but here we do something approaching with BREs by doing \(a\{0,1\}b\{0,1\}\)* ((a?b?)* in ERE) as an equivalent of (a|b)*. Using (a*b*)* as in Rakesh's answer would also work.

grep would not be an option as standard grep only prints the full matching lines. awk uses EREs though. Standard awk doesn't have capture groups, but you should be able to do things like:

awk "match(\$0, /^([^'\"#]|\"[^\"]*\"|'[^']*')*#/) {
       \$0 = substr(\$0, 1, RLENGTH-1)
     }
     {print}"

With your edited requirements, you can handle the escaped quotes by using "(\\.|[^\\"])*" or its BRE equivalent:

sed 's/^\(\(\([^"\\'\''#]\)*\(\\.\)\{0,1\}\("\([^"\\]*\(\\.\)\{0,1\}\)*"\)\{0,1\}\('"'[^']*'\)\{0,1\}\)*\)#.*/\1/"

or:

awk 'match($0, /^([^'\''"\\#]|\\.|"(\\.|[^\\"])*"|'\''(\\.|[^\\'\''])*'\'')*#/) {
       $0 = substr($0, 1, RLENGTH-1)
     }
     {print}'

both of which also handle escaped quotes outside of quotes (as in foo\"bar # comment).

I've switched to using single quotes here to reduce the number of backslashes that need to be inserted to get a literal \\ , but that literal single quotes in the data have to be inserted as 'before'\''after', that is '\'', the first ' to close the 'before' quoted strings, \' using backslash to quote/escape the literal ' (as you can't insert a single quote inside a single-quoted string) and the 'after' quoted string follows.

Answer 2

Based on the rules specified we distinguish 5 kinds of words:

• double quoted words (they can include escaped double quotes as well) "... \"... "

• single quoted words '...' they will not have an included single quote.

• backslash quoted word \.basically any escaped char.

• non comment starting char [^'#"]

• what remains is a comment.

#! /bin/bash
# whitespace and horizontal whitespace
_ws_=$(printf '\t \nx') 
ws="[${_ws_%?}]" hws="[${_ws_%??}]"
nac="[^\"'#]" nac="($nac)" #not a comment char
bqw='[].'    bqw="($bqw)" # backslashed word
sqw="'[^']*'" sqw="($sqw)" # single quoted word
#double quoted word 
dqw='
  "
    (
      [^\"]* ([][])* []"
    )*
    [^"]*
  "
'
dqw="(${dqw//$ws/})"
sed 

  -e '/#/!b' 

  -e "s/^(($sqw$dqw$bqw$nac))./\1/" 

  -e "s/$hws*$//" 

< file

Note this is fully POS IX

Answer 3

Solution

The following solution works with popular sed implementations like GNU sed which support extended regular expressions (ERE):

sed -E "s/^(([^#\"'\\]|'[^']*'|\"([^\"\\\\]|\\\\.)*\")*)#.*/\1/" input.txt

The main advantage of this solution is better readability than many other solutions.

Note: The -E switch is not part of POSIX 2018, yet, but it is on its way to become part of become POSIX 2020. If you need a POSIX-2018-compatible solution see Stéphane Chazelas' answer.

How it works

The following longer version breaks the above regex into pieces that are easier to digest:

NON_QUOTED_TEXT="[^#\"'\\]"
SINGLE_QUOTED_STRING="'[^']*'"
DOUBLE_QUOTED_STRING='"([^"\\]|\\.)*"'
REMOVE_COMMENTS="^((${NON_QUOTED_TEXT}|${SINGLE_QUOTED_STRING}|${DOUBLE_QUOTED_STRING})*)#.*"
sed -E "s/${REMOVE_COMMENTS}/\1/" input.txt

We are using sed to search for text matching the regular expression contained in ${REMOVE_COMMENTS} and replace each match with the content of the first capture group \1. This capture group contains the match of the regular expression between the first opening parenthesis ( and the last closing parenthesis ). This part of the regex matches any text before the first comment sign (#) which doesn't occur as part of a quoted string. Looking at it in detail we are matching a sequence of 0 to N (*) of the following options (a|b|c):

• Non-quoted text: characters other than #, ", ' and \.
• Single-quoted text: any number (*) of characters other (^) than ' enclosed by a pair of single quotes.
• Double-quoted text: A string enclosed by a pair of double quotes. The string is allowed to contain any number of characters other than " and \ or ((a|b)) an arbitrary character preceded by a backslash (\\.).

When you're combining the parts to the complete solution above we have to keep in mind that Bash rules require slightly different quoting when using single vs. double quotes. See Differences between single and double quotes in Bash for a details.

Answer 4

command

 sed -e '/^#/d' filename| sed "s/# comment$//g"

Python

#!/usr/bin/python
import re
d=re.compile(r'^#')
r=re.compile(r'#\scomment$')
l=open('p','r')
for  i in l:
    if not re.search(d,i):
        e=re.sub(r,"",i)
        print e.strip()

output

some text # with a comment
some "quoted text # not a comment"
'# not a comment' "# it's not a comment" '#still not a comment

'