6

In Bash, piping tail -f to a read loop blocks indefinitely. 

while read LINE0 
do 
    echo "${LINE0}"; 
done < <( tail -n 3 -f /tmp/file0.txt | grep '.*' ) 
# hangs

Remove the -f or | grep '.*', then the loop will iterate.

The following does not hang.

tail -n 3 -f /tmp/file0.txt | grep '.*'

What causes this behavior?

Is there anyway in Bash to follow a file and read in a pipe expression?

Eric Larson
  • 85
  • If you use -f, it is supposed to "hang" in the sense that tail keeps waiting for more lines to be added to the file. When I try tail -n 3 -f /tmp/file0.txt | wc, it hangs for me. It should hang because wc never gets an end-of-file signal. Are you sure that it doesn't hang for you? – John1024 Sep 12 '16 at 21:14
  • That's a good point. I corrected my question, replacing wc with grep. – Eric Larson Sep 12 '16 at 21:18
  • grep may not "hang" but it does have buffering issues. It may not print anything until there is enough to fill a buffer. – John1024 Sep 12 '16 at 21:21
  • Thanks @John1024! I demonstrated that you are correct by: I added lines to the file, and the loop finally iterated. It looks like the buffering is not in the tail -f, but in the read. If I grep for an expression that is only in a couple of lines, even if the file is grown large, the loop will not iterate. I now need to figure out how to stop read from buffering. – Eric Larson Sep 12 '16 at 21:34

2 Answers2

18

In the pipeline, grep's output is buffered. With the GNU implementation of grep, you can force flush the output after each line with --line-buffered (documentation here); for example:

tail -n 3 -f /tmp/file0.txt | grep --line-buffered '.*' |
  while IFS= read -r LINE0 
  do 
    printf '%s\n' "${LINE0}"
  done
Stéphane Chazelas
  • 544,893
Ipor Sircer
  • 14,546
  • 1
  • 27
  • 39
1

Greg's Wiki has a comprehensive article about buffering: https://mywiki.wooledge.org/BashFAQ/009

The technique that worked for me (Popos 20.04, Ubuntu 20.04) was using stdbuf to do line buffering. Their example:

tail -f logfile | stdbuf -oL grep 'foo bar' | awk ...

My use case:

journalctl --output=json -t poke-stats -f |\
 stdbuf -oL jq -r '.__REALTIME_TIMESTAMP' |\
 stdbuf -oL awk '{print $1-last;last=$1}'
Max Murphy
  • 121