/bin/sh evaluates string while Bash does not

Question

When running the script below, I get two different outputs, depending on whether or not the shell used is sh or bash:

regex(){
     echo 's/\(.* \)\(!\{0,1\}\)has(/\1\2MOCK_has(/g'
}
replace_builtins(){
    sed -e "$(regex)"
}
echo 'if !has("nvim"): ' | replace_builtins

• bash: if !MOCK_has(\"nvim\"):
• sh: ??MOCK_has(\"nvim\"):

(those question marks were originally copied verbatim from the terminal, but disappeared when I saved the post. It's essentially non-printable characters)

I would like to know what is happening when running in POSIX sh mode that explains this phenomena.

EDIT: for bonus points explain why this also happens in Bash when substituting echo for printf in the regex function:

     printf 's/\(.* \)\(!\{0,1\}\)has(/\1\2MOCK_has(/g'

Answer 1

The explanation is in the POSIX specification for echo:

A string to be written to standard output. If the first operand is -n, or if any of the operands contain a <backslash> character, the results are implementation-defined.

POSIX mostly codified historical practice, and sometimes historical practice was not consistent. Some shells expand escape sequences in the arguments to echo, for example \t expands to a tab and \1 expands to the character with byte value 1 (^A). Other shells treat backslash as an ordinary character.

The portable way to print an arbitrary string is with printf. printf always expands backslash escape sequences in its first argument (the format). To print a string literally, use

printf %s 's/\(.* \)\(!\{0,1\}\)has(/\1\2MOCK_has(/g'

To print a string literally and add a newline at the end, use

printf '%s\n' 's/\(.* \)\(!\{0,1\}\)has(/\1\2MOCK_has(/g'

Note that if the string is written in the shell script using a single-quoted literal, a single quote character needs to be written as '\'''. This is about the shell syntax, a completely different problem from printing a string literally.

Answer 2

You identifed an interesting problem...

The problem is missing POSIX compatibility in dash.

POSIX distincts bewteen a basic POSIX compatibility level for tiny embedded systems and larger systems (like e.g. Linux) that claim UNIX compatibility. In the latter case, the system needs to implement all so called XSI extensions.

An XSI compatible system would need to expand certain backslash escapes in echo arguments.

bash can be compiled to behave POSIX/XSI compliant (and this is e.g. done on Solaris and MacOS), but this is not done for the bash binary on Linux. If bash is compiled for POSIX/XSI compliance, it correctly handles backslash escapes for echo arguments and your example code would work with such a bash binary from Solaris or MacOS, because there is no POSIX/XSI escape sequence in your example code.

Since bash on Linux is not XSI compliant, it does not expand backlash escapes for echo arguments at all and this is why your example code works with bash on Linux as well.

dash on the other side claims POSIX/XSI compliance and expands backslash escapes for echo arguments. If dash did implement POSIX/XSI compliance correctly, your example code did work with dash as well. This is because your example code does not contain any POSIX/XSI backslash escape sequence.

POSIX/XSI requires echo to expand:

\0nnn  for an octal number that represents the related character

Your example code contains the backslash sequences:

\1 for the first sed subexpression

and

\2 for the second sed subexpression

and this is not part of the POSIX/XSI echo escape sequences, so the builtin echo from a POSIX compliant shell is not permitted to expand them. dash however incorrectly expands \1 and \2 as octal numbers even tough this is forbidden by POSIX. This is why your example code fails with dash.

I recommend you to make a bug-report against dash and either wait for a fix, or to replace echo arg by printf '%s\n' arg. This works even with dash because the known bug with the builtin printf in dash does not affect your case.

So we can list the POSIX/XSI bugs from dash as:

• does not support multy-byte characters.

• expands \nnn in echo arguments even though this is forbidden

• does not expand \nnn in printf arguments even though this is required.