Can't pipe in bash's "mapfile" ... but why?

Question

I just want to get all the files in a certain directory into a bash array (assuming that none of the files have a newline in the name):

So:

myarr=()
find . -maxdepth 1  -name "mysqldump*" | mapfile -t myarr; echo "${myarr[@]}"

Empty result!

If I do the roundabout way of using a file, temporary or otherwise:

myarr=()
find . -maxdepth 1  -name "mysqldump*" > X
mapfile -t myarray < X
echo "${myarray[@]}"

Result!

But why doesn't mapfile read properly from a pipe?

Answer 1

From man 1 bash:

Each command in a pipeline is executed as a separate process (i.e., in a subshell).

Such subshells inherit variables from the main shell but they are independent. This means mapfile in your original command operates on its own myarr. Then echo (being outside the pipe) prints empty myarr (which is the main shell's myarr).

This command works differently:

find . -maxdepth 1 -name "mysqldump*" | { mapfile -t myarr; echo "${myarr[@]}"; }

In this case mapfile and echo operate on the same myarr (which is not the main shell's myarr).

To change the main shell's myarr you have to run mapfile in the main shell exactly. Example:

myarr=()
mapfile -t myarr < <(find . -maxdepth 1 -name "mysqldump*")
echo "${myarr[@]}"

Answer 2

Bash runs the commands of a pipeline in a subshell environment, so any variable assignments etc. that take place within it aren't visible to the rest of the shell.

Dash (Debian's /bin/sh) as well as busybox's sh are similar, while zsh and ksh run the last part in the main shell. In Bash, you can use shopt -s lastpipe to do the same, but it only works when job control is disabled, so not in interactive shells by default.

So:

$ bash -c 'x=a; echo b | read x; echo $x'
a
$ bash -c 'shopt -s lastpipe; x=a; echo b | read x; echo $x'
b

_{(read and mapfile have the same issue.)}

Alternatively (and as mentioned by Attie), use process substitution, which works like a generalized pipe, and is supported in Bash, ksh and zsh.

$ bash -c 'x=a; read x < <(echo b); echo $x'
b

POSIX leaves it unspecified if the parts of a pipeline run in subshells or not, so it can't really be said that any of the shells would be "wrong" in this.

Answer 3

As Kamil has pointed out, each element in the pipeline is a separate process.

You can use the following process substitution to get find to run in a different process, with the mapfile invocation remaining in your current interpreter, allowing access to myarr afterwards:

myarr=()
mapfile -t myarr < <( find . -maxdepth 1  -name "mysqldump*" )
echo "${myarr[@]}"

b < <( a ) will act similarly to a | b in terms of how the pipeline is wired - the difference is that b is executed "here".

Answer 4

If we use:

myarr=()
mapfile -t myarr < <( find . -maxdepth 1  -name "mysqldump*" )
echo "${myarr[@]}"

${myarr[@]} is not lost but $? of <(find..) is lost

If we use:

find . -maxdepth 1 -name "mysqldump*" | mapfile -t myarr

${myarr[@]} is lost but $? of find is not lost

if we use

shopt -s lastpipe
find . -maxdepth 1 -name "mysqldump*" | mapfile -t myarr

${myarr[@]} is not lost, $? of find is not lost but you need to disable bash job control (set +m)

if we use something like this

line=$(find . -maxdepth 1 -name "mysqldump*"); rc=$?; mapfile -d '\n' myarr <<< "$line"

${myarr[@]} is not lost, $? of find is not lost, bash job control (set -m) does not need to be disabled ... but you you can't use \0 null byte as delimiter (find ... -print0 | mapfile -d '' ...)

The only way I have found come from https://stackoverflow.com/questions/51072044/process-substitution-and-exit-codes and need bash 4.4.x

exec {psfd}< <(find . -maxdepth 1 -name "mysqldump*" -print0); procsub_pid=$!
mapfile -d '\0' myarr <&$psfd
exec {psfd}<&-
wait "$procsub_pid"
echo $?
echo "${myarr[@]}"

${myarr[@]} is not lost, $? of find is not lost, bash job control (set -m) does not need to be disabled and \0 null byte can be used as delimiter.

Answer 5

I don't know why, per se, but looking at shellcheck SC2206: Quote to prevent word splitting/globbing, or split robustly with mapfile or read -a., I can see how to make your example work: use a herestring (<<<), like this:

# For bash
mapfile -t array <<< "$var"

For your particular case, therefore, do this:

# Capture all `find` output into a regular "indexed" bash array by using
# `mapfile` and a herestring (`<<<`)
mapfile -t array <<< "$(find . -maxdepth 1  -name "mysqldump*")"
Print all elements at once in a big blob (this also demonstrates how
to pass all elements in the array as arguments to another function)
echo "${array[@]}"
OR, print all elements nicely, one element per line
for element in "${array[@]}"; do
    echo "  $element"
done
OR, the same thing as just above but as a one-liner:
for element in "${array[@]}"; do echo "  $element"; done

Answer 6

Here I am presenting my solution.

I refer to items in the very usefull arcticle

cant-pipe-in-bashs-mapfile-but-why

in

superuser: /questions/1348948/

There the main goal is not only to store the filenames in the array, but to store each one line as an alement in the array.

This I have solved for me.

Here I present my current solution. It is a full-featured script.

• the array filled by bash command mapfile, resp. readarray is not lost;

• $? of find and of mapfile are printed (of find to STDERR);

• bash job control (set -m) does not need to be disabled; thus the script can - without any preparatory work - be run out of "fc";

• the problems with the delimiter ascii null are avoided;

Remarks:

The script is in a debugging state:

lines are numbered with the index of the array;

before lines of a new file will begin, filename with complete directory entry is put into an element of the array and is printed;

framed with comment lines;

both, the directory entry and the framing lines have "## " at beginning of line;

thus can easily be parsed and removed;

# +++++
mapfile -t  WEarr  <<< "$(find /tmp -maxdepth 1 -name "WEx_08*"  -exec /usr/bin/printf "#%.0s" {1..70} ; -exec echo "" ; -exec echo -n "## " ; -exec ls -lad --time-style=long-iso {} ;  -exec /usr/bin/printf "#%.0s" {1..70} ; -exec echo "" ;   -exec cat {} ; ; WE_retcod_find=$?; echo "WE_retcod_find: ~$WE_retcod_find~" >&2)"
WE_retcod_mapfile=$?
echo "WE_retcod_mapfile: $WE_retcod_mapfile~"
WE_noe_WEarr=${#WEarr[@]}
echo "WE_noe_WEarr: ~$WE_noe_WEarr~"
for WE_element in "${WEarr[@]}"; do
echo "WE_element: ~$WE_element~"
done
for ((i=0; $i<$WE_noe_WEarr; i++)); do
  echo "i: $i ;; ~${WEarr[$i]}~"
done
-----

-.-