Running up to X commands in parallel

Question

I'm running something like this:

find . -maxdepth 1 -type f -note -iname "*.gpg" | sort | while read file ; do
    echo "Encrypting $file..."
    gpg --trust-model always --recipient "me@myself.com" --output "$file.gpg" \
        --encrypt "$file" && rm "$file" 
done

This runs great, but it seems that GPG is not optimized to use multiple cores for an encryption operation. The files I'm encrypting are about 2GB in size and I have quite a bit of them. I'd like to be able to run X jobs in parallel to encrypt the files and then remove them. How can I do this, setting a limit to, say, 8 jobs at a time?

score 5 · Accepted Answer · edited May 23 '17 at 12:40

5

If you install the GNU Parallel tool you can make pretty easy work of what you're trying to accomplish:

$ find . -maxdepth 1 -type f -note -iname "*.gpg" | sort | \
      parallel --gnu -j 8 --workdir $PWD '                 \
         echo "Encrypting {}...";                          \
         gpg --trust-model always                          \
           --recipient "me@myself.com" --output "{}.gpg"   \
           --encrypt "{}" && rm "{}"                       \
      '

details

The above is taking the output of find and running it through to parallel, and running 8 at a time. Everywhere there's an occurrence of {} the filenames that are being passed through from find will replace the {} in those spots.

References

Running shell script in parallel

edited May 23 '17 at 12:40

Community

1

answered Jun 24 '13 at 17:25

slm

369,824

What's with the hanging apostrophe at the end? Is it missing a starting apostrophe? – Naftuli Kay Jun 24 '13 at 17:43
@TKKocheran - did that work for you? – slm Jun 24 '13 at 17:55
This is what I'm running, and this is the output I'm seeing. Not working yet :-| – Naftuli Kay Jun 24 '13 at 18:08
What if you move the sudo outside of the find? Make it a sudo -i find ... I don't think you can pipe to sudo in that manner. You could do this: echo "cmd1" | sudo bash. – slm Jun 24 '13 at 18:20
1

Looks like I got pwnt with this problem. – Naftuli Kay Jun 24 '13 at 18:32
@TKKocheran - Ha. happens to the best of us 8-). I shoot so that it's only once a day ... it's never been once a day 8-). What's your distro? I had to download and re-compile parallel for my Fedora system. – slm Jun 24 '13 at 18:34
Ok, so now I'm getting gpg: can't open \': No such file or directory. What the heck? It seems that it's not replacing the{}`. – Naftuli Kay Jun 24 '13 at 18:36
Ah, had to remove --semaphore. – Naftuli Kay Jun 24 '13 at 18:37
I used this command as a model and it shows 2 lines for each file: find . | parallel -j 8 --workdir $PWD 'echo {}; echo {}'. – slm Jun 24 '13 at 18:38
This is one heck of a way to load test. Let's cook some cores! – Naftuli Kay Jun 24 '13 at 18:39
@TKKocheran - I'd never tried this command till today, thanks for the question, now I know how to do this too! – slm Jun 24 '13 at 18:40

score 4 · Answer 2 · edited Jun 11 '20 at 12:04

You might want to look at gnu parallel and its --semaphore option. From the documentation:

--semaphore

Work as a counting semaphore. --semaphore will cause GNU parallel to start command in the background. When the number of simultaneous jobs is reached, GNU parallel will wait for one of these to complete before starting another command.

You use --jobs 8 to limit the number of jobs to 8. You can pipe the output of sort into parallel like you would do with xargs. sem is an alias for parallel --semaphore

score 0 · Answer 3 · answered Feb 01 '19 at 11:11

0

I have written an easy-to-use Perl script that allows to control the maximum number commands that are run at the same time: https://github.com/matmu/parallelize_cmds

Might be of interest for you.

answered Feb 01 '19 at 11:11

Matthias Munz

101

Running up to X commands in parallel

3 Answers3

details

References

Linked