2

I am learning GNU parallel and I wonder it this makes sense:

bash:

    IFS=" "
while read field || [ -n "$field" ]; do
    targets_array+=("$field")

done </location/targets


parallel  -Dall bash myScript.sh ::: $targets_array

I wonder if it makes sense because my output seem to stop at some point... I have 30.000 targets that I scan with myScript.sh then I update info about them in DB also using myScript.sh

I tried to use some options but I could not make it work: like writing to a logfile from the performance point of view, does it make sense to run one target at the time?

dwt.bar
  • 145
  • 1
  • 10

1 Answers1

4

$targets_array is equivalent to ${targets_array[0]}. To get all elements you need ${targets_array[@]}. And you should quote right.

So it could be:

parallel … ::: "${targets_array[@]}"               # but don't

parallel is an external command. If the array is large enough then you will hit argument list too long. Use this instead:

printf '%s\n' "${targets_array[@]}" | parallel …   # still not the best

It will work better because in Bash printf is a builtin and therefore everything before | is handled internally by Bash.

I notice you didn't use read -r (I assume it was an educated decision), so a backslash-newline pair (if any) in /location/targets can result in a newline character actually in some array element. Therefore separating with newlines while passing data to parallel may be a bug. Separate with null bytes:

printf '%s\0' "${targets_array[@]}" | parallel -0 …

Hopefully /location/targets does not contain null bytes. If it does then they won't get to the array in the first place.

Kamil Maciorowski
  • 21,864
  • 2
    I don't really see the point of the array at all actually. Why not just read targets directly with parallel? – Kusalananda Dec 02 '20 at 08:11
  • 1
    @Kusalananda Because I assume read without -r was an educated decision. I'm not aware of GNU parallel's ability to handle backslash equivalently. Can it do this? – Kamil Maciorowski Dec 02 '20 at 08:15
  • 1
    I assumed that this was an omission in the question. Setting IFS to a space character indicates that they either don't know what this does (and that omitting -r was a mistake), or that they explicitly want to trim flanking spaces from the input data (in which case omitting -r may have been on purpose also). Unfortunately, the question does not tell. – Kusalananda Dec 02 '20 at 08:31
  • @Kusalananda The answer is now community wiki. If you think it should elaborate on the lack of -r or provide a command to read targets directly with parallel then please improve it. Unfortunately I have no time for this now. – Kamil Maciorowski Dec 02 '20 at 08:39
  • @Kusalananda It looks like the omission of -r was a mistake, I tried to set the field separator as a space and read target after target separated by space from the file. – dwt.bar Dec 02 '20 at 17:01
  • @KamilMaciorowski I tried to cat the file and It would not work, it treated my targets as a command, I will come up with some good example if I have some more time. Thank you for your contribution, It was very helpful. – dwt.bar Dec 02 '20 at 17:03