How to avoid a useless use of cat when parsing a file?

Question

I have a file containing a list of directories. For instance

/foo/bar/dir1
/foo/bar/dir2
/foo/bar/dir3

I want to create all these directories. Here's what I did:

for dir in $(cat myfile); do mkdir $dir; done

What would be the correct way of doing this while avoiding the "useless use of cat"?

Ideally, answers would focus on Ksh88, but I'm also interested in other shells

if all the directories are under single directory and are few then use brace expansion of shell. $ mkdir /foo/bar/dir{1,2,3} or $ mkdir /foo/bar/{dir1,dir2,dir3} — Nikhil Mulley, Dec 19 '11 at 19:39
possible duplicate of How can I read line by line from a variable in bash? — Gilles 'SO- stop being evil', Dec 19 '11 at 23:33
Thanks Gilles for pointing out that question, I was not aware of it. For the record, I was looking to read from a file not a variable. And my question is shell-agnostic (if anything I mention ksh). In any case I found my answer; I was not familiar with read. The man page will do the rest. — rahmu, Dec 20 '11 at 13:54

score 13 · Accepted Answer · edited Dec 20 '11 at 00:47

13

AProgrammer's suggestion of using xargs is often best, but another option is to use redirection into a while loop, which allows additional commands to be made and variables to be set:

while read -r dir; do mkdir $dir; done < myfile

An example of a more complicated structure would be:

now=`date +%Y%m%d.%H%M%S`
while read -r dir; do
    newdistfile="/tmp/dist-`echo $dir | tr / _`.tgz"
    mv $dir ~/backups/$dir.$now &&
        mkdir $dir &&
        tar xzfC $newdistfile $dir
done < myfile

This is not something that xargs could do without writing a 'helper program'.

edited Dec 20 '11 at 00:47

Kevin

40,767

answered Dec 19 '11 at 16:29

Arcege

22,536

What are the limits of using xargs over read? It is my understanding that xargs will fork a new process (unlike read that is a shell builtin). Isn't this something you want to avoid? – rahmu Dec 19 '11 at 16:40
xargs will operate on a single command, so if you want to have a compound statement (like more than one, an if, while or for, etc.) or setting a variable, then you cannot use xargs. Yes, xargs will fork one new statement, plus one for some number of input lines, which is very little overhead. However, some older versions of xargs do not take a -0 option, so a while read will be the way to go if there may be whitespace in the input directory names. – Arcege Dec 19 '11 at 17:38

AProgrammer · Answer 2 · 2011-12-19T16:39:48.083

12

I'd do something like

xargs mkdir < myfile

edited Dec 19 '11 at 16:39

answered Dec 19 '11 at 16:04

AProgrammer

2,318

1

Why "-n 1"? You can do several directories with one invocation of mkdir. – ddeimeke Dec 19 '11 at 16:29
1

@ddeimeke, because I didn't remember if it is POSIX or just a common extension. Now I've checked and it is in SUSv2. – AProgrammer Dec 19 '11 at 16:39
3

Useless use of xargs. ;) – user unknown Dec 19 '11 at 19:25
Note that this assumes that the directory names are quoted in the input file if they contain whitespaces. – Kusalananda Feb 22 '22 at 18:08

user unknown · Answer 3 · 2011-12-19T19:24:03.710

12

At least in bash, as long as there are no filenames containing spaces and newlines, this:

mkdir $(< myfile)

works. So we have a useless use of for, xargs too.

< does not start a new process in bash, in contrast to cat, but I don't know for ksh.

edited Dec 19 '11 at 19:24

answered Dec 19 '11 at 19:15

user unknown

10,482

2

+1 for clarifying, as opposed to other answer, the spaces problem. – enzotib Dec 19 '11 at 20:05

jlliagre · Answer 4 · 2011-12-19T22:41:17.680

4

This is the solution closer to your original script still avoiding the useless cat

for dir in $(<myfile); do mkdir $dir; done

It still contains a useless loop though.

edited Dec 19 '11 at 22:41

answered Dec 19 '11 at 17:07

jlliagre

61,204

score 3 · Answer 5 · answered Dec 19 '11 at 21:09

I wouldn't say that's a useless use of cat.

In the classic sense, a UUOC means writing:

cat file | some_command and its args ...

instead of the equivalent and cheaper

<file some_command and its args ...

or (equivalently and more classically)

some_command and its args ... < file

In this case:

for dir in $(cat myfile); do mkdir $dir; done

there is no command into which the output of cat is being piped.

There are certainly alternatives that don't use cat. In particular, for bash and ksh $(<myfile) is likely to be faster than $(cat myfile), though not all shells support that construct. But I wouldn't call the original code a "useless use of cat" any more than I'd call AProgrammer's solution a "useless use of xargs".

score 1 · Answer 6 · answered Oct 20 '16 at 22:20

1

With bash 4 and limited by ARG_MAX

mapfile -t <file && mkdir -- "${MAPFILE[@]}"

answered Oct 20 '16 at 22:20

iruvar

16,725

or zmodload zsh/mapfile && mkdir -- ${(f)mapfile[infile]} if the shell is zsh – don_crissti Oct 20 '16 at 22:34

Kajukenbo · Answer 7 · 2022-02-22T18:02:41.887

Yours is a perfect example of intended POSIX usage of a while-loop, specifically what I call a "While Redirect".

"While Redirect"

while read dir
do
  mkdir $dir
done < myfile

Now, compare the previous example to what I call a "While Pipe":

"While Pipe"

cat myfile | while read dir
do
  mkdir $dir
done

The cat is not an UUoC in the While Pipe; it is part of the command syntax. Just like you might have used grep -E . myfile | while or something else.

It is considered Useless in the for-loop mostly because it --in conjunction with the backquotes-- is considered to be inefficient and dangerous there.

See here for more details

How to avoid a useless use of cat when parsing a file?

7 Answers7