2

I'm using the below command in my script to perform a dos2unix operation on the files which I copy from Windows to Linux. There are many files, so it takes a considerable amount of time to perform this operation.

I searched on the internet for optimizing this find and found that we can use xargs with find instead of -exec to increase the performance, but I'm struggling to convert the below one to use with xargs:

find /path_to_files/ -exec bash -c 'dos2unix -k -n "{}" tmp_file && mv tmp_file "{}"' \;
Jeff Schaller
  • 67,283
  • 35
  • 116
  • 255
Ashish
  • 23

2 Answers2

8

It is actually much more simple. You do not need to invoke a shell nor use mv because dos2unix already modifies the target file if you do not give it the -n flag.

find /path_to_files -type f -exec dos2unix -k -q -- {} +

-type f so that find only searches for regular files, not directories. -q flag is given to dos2unix so that it does not write informational messages to stdout. And we use {} + instead of {} \; so that find does not invoke one dos2unix process for each file found, instead it gives so much files as it can as arguments (dos2unix file1 file2 file3 ...).

Xargs would just be one extra process and thus one extra time penalty. Avoid invoking new processes whenever you can.

Quasímodo
  • 18,865
  • 4
  • 36
  • 73
1

Create a script in ~/bin to process a list of files, call it doit, containing

#!/bin/bash
while [[ $# -ne 0 ]] ; do
    thisfile="$1"
    shift
    dos2unix -k -n "$thisfile" tmp_file
    mv tmp_file "$thisfile"
done

Then,

chmod +x ~/bin/doit

And finally,

find /path/to/files -type f -print0 | \
    xargs -0 -r $HOME/bin/doit

Read man find xargs bash.

waltinator
  • 4,865
  • @waltinator Instead of the while loop with assignment and shift, you could simply write for thisfile in "$@"; do ... or even shorter for thisfile; do ... – the latter automatically loops over the command line arguments. – DonHolgo Aug 23 '20 at 17:47
  • @Ashish This should work for a million files. xargs knows how many args the command line can contain. It reruns the doit command as many times as it needs for the total number of files. – Paul_Pedant Aug 23 '20 at 17:51