sed parameter substitution with multiline quoted string

Question

Goals

Replace the text "scripts: {" with the following string

"scripts": {
    "watch": "tsc -w",

in a json file.

Attempts

I created two variables for source and destination strings:

First attempt

SRC='"scripts": {'
DST='"scripts": {
    "watch": "tsc -w",'

And ran the following command:

sed "s/$SRC/$DST/" foo.json

This has failed.

Second attempt

This time I escaped double quotes for the source and destination strings:

SRC="\"scripts\": {"
DST="\"scripts\": {
    \"watch\": \"tsc -w\",
    \"dev\": \"nodemon dist/index.js\","

And ran the same command as above, which failed.

Third and fourth attempts

I tried the variables defined as above with the following command:

sed 's/'"$SRC"'/'"$DST"'/' foo.json

This has failed.

All these attempts yielded the error

unterminated 's' command

What has gone wrong?

Answer 1

Assuming your JSON document looks something like

{
  "scripts": {
    "other-key": "some value"
  }
}

... and you'd like to insert some other key-value pair into the .scripts object. Then you may use jq to do this:

$ jq '.scripts.watch |= "tsc -w"' file.json
{
  "scripts": {
    "other-key": "some value",
    "watch": "tsc -w"
  }
}

or,

$ jq '.scripts += { watch: "tsc -w" }' file.json
{
  "scripts": {
    "other-key": "some value",
    "watch": "tsc -w"
  }
}

Both of these would replace an already existing .scripts.watch entry.

Note that the order of the key-value pairs within .scripts is not important (as it's not an array).

Redirect the output to a new file if you want to save it.

To add multiple key-value pairs to the same object:

$ jq '.scripts += { watch: "tsc -w", dev: "nodemon dist/index.js" }' file.json
{
  "scripts": {
    "other-key": "some value",
    "watch": "tsc -w",
    "dev": "nodemon dist/index.js"
  }
}

In combination with jo to create the JSON that needs to be added to the .scripts object:

$ jq --argjson new "$( jo watch='tsc -w' dev='nodemon dist/index.js' )" '.scripts += $new' file.json
{
  "scripts": {
    "other-key": "some value",
    "watch": "tsc -w",
    "dev": "nodemon dist/index.js"
  }
}

---

sed is good for parsing line-oriented text. JSON does not come in newline-delimited records, and sed does not know about the quoting and character encoding rules etc. of JSON. To properly parse and modify a structured data set like this (or XML, or YAML, or even CSV under some circumstances), you should use a proper parser.

As an added benefit of using jq in this instance, you get a bit of code that is easily modified to suit your needs, and that is equally easy to modify to support a change in the input data structure.

Answer 2

Kusalananda is absolutely right saying that a dedicated parser is the right tool for the job. However, this can easily be done in sed (at least with GNU sed which understands \n) as well. You were making things more complicated by trying to replace the entire two-line pattern. Instead, just match the target string and insert the replacement after it:

"scripts": {

And then:

$ sed '/"scripts": {/s/$/\n    "watch": "tsc -w",/' file
"scripts": {
    "watch": "tsc -w",

This means "if this line matches the string "scripts": {, then replace the end of the line with a newline (\n) followed by "watch": "tsc -w",. Alternatively, you can use the a sed command to append text:

$ sed '/"scripts": {/a\    "watch": "tsc -w",' file 
"scripts": {
    "watch": "tsc -w",

Answer 3

Here, we create a Sed command, but put a newline into it:

sed "s/$SRC/$DST/" foo.json

That's invalid, as the newline ends the command. We need to write \n instead of a literal newline. In some shells (Bash, zsh, others..., but not plain POSIX shell), we can do that using a parameter substitution:

DST=${DST//$'\n'/\\n}
#        //             replace every
#          $'\n'        newline
#               /       with
#                \\n    \ and n 

For other replacement strings, we may also need to quote /, too:

DST=${DST//\//\\/}

Also, \ (do that one first) and & have special meaning in replacement text and therefore would need replacement.

Answer 4

While I agree that jq is the optimum tool to use, this particular task can easily be done using sed. Essentially, the task is to append a line of text after a pattern on the previous line is matched.

$ cat example.json
{
   "scripts": {  
      "access_token": "asadasd",
      "expires_in": "3600"
   }
}
$ cat demo.sh
#!/bin/bash
SRC='"scripts": {'
DST='\ \ \ \ \ \ "watch": "tsc -w",'
sed '/'"$SRC"'/a '"$DST"'' example.json
$ ./demo.sh
{
   "scripts": {

      "watch": "tsc -w",
      "access_token": "asadasd",
      "expires_in": "3600"
   }
}
$

Answer 5

Since sed regards all text as regex, so we have to escape strings before we plug them into sed code. Plus, we need to take care whether the string going in is on the lhs or rhs of an s/// command, coz the list of BRE chars are different on both sides.

s='[[:blank:]]'
srch_str='"scripts": {'
repl_str='"watch": "tsc -w",'
esc_srch_str=$(
  printf '%s\n' "$srch_str" |
  sed -e 's:[][/.^$*]:\&:g'
)
esc_repl_str=$(
  printf '%s\n' "$repl_str" |
  sed -e 's:[/&]:\&:g'
)
sed -e "
  /^$s$esc_srch_str$/G
  s/^($s).*\n/&\1  $esc_repl_str/
" file

Output:

{
  "scripts": {
    "watch": "tsc -w",
    "other-key": "some value"
  }
}