GNU parallel bash

Question

I am learning GNU parallel and I wonder it this makes sense:

bash:

    IFS=" "
while read field || [ -n "$field" ]; do
    targets_array+=("$field")
    
done </location/targets
 
parallel  -Dall bash myScript.sh ::: $targets_array

I wonder if it makes sense because my output seem to stop at some point... I have 30.000 targets that I scan with myScript.sh then I update info about them in DB also using myScript.sh

I tried to use some options but I could not make it work: like writing to a logfile from the performance point of view, does it make sense to run one target at the time?

score 4 · Accepted Answer · answered Dec 02 '20 at 08:08

4

$targets_array is equivalent to ${targets_array[0]}. To get all elements you need ${targets_array[@]}. And you should quote right.

So it could be:

parallel … ::: "${targets_array[@]}"               # but don't

parallel is an external command. If the array is large enough then you will hit argument list too long. Use this instead:

printf '%s\n' "${targets_array[@]}" | parallel …   # still not the best

It will work better because in Bash printf is a builtin and therefore everything before | is handled internally by Bash.

I notice you didn't use read -r (I assume it was an educated decision), so a backslash-newline pair (if any) in /location/targets can result in a newline character actually in some array element. Therefore separating with newlines while passing data to parallel may be a bug. Separate with null bytes:

printf '%s\0' "${targets_array[@]}" | parallel -0 …

Hopefully /location/targets does not contain null bytes. If it does then they won't get to the array in the first place.

answered Dec 02 '20 at 08:08

Kamil Maciorowski

19,242
1
50
94

2

I don't really see the point of the array at all actually. Why not just read `targets` directly with `parallel`? – Kusalananda Dec 02 '20 at 08:11
1

@Kusalananda Because I assume `read` without `-r` was an educated decision. I'm not aware of GNU `parallel`'s ability to handle backslash equivalently. Can it do this? – Kamil Maciorowski Dec 02 '20 at 08:15
1

I assumed that this was an omission in the question. Setting `IFS` to a space character indicates that they either don't know what this does (and that omitting `-r` was a mistake), or that they explicitly want to trim flanking spaces from the input data (in which case omitting `-r` may have been on purpose also). Unfortunately, the question does not tell. – Kusalananda Dec 02 '20 at 08:31
@Kusalananda The answer is now community wiki. If you think it should elaborate on the lack of `-r` or provide a command to read `targets` directly with `parallel` then please improve it. Unfortunately I have no time for this now. – Kamil Maciorowski Dec 02 '20 at 08:39
@Kusalananda It looks like the omission of -r was a mistake, I tried to set the field separator as a space and read target after target separated by space from the file. – dwt.bar Dec 02 '20 at 17:01
@KamilMaciorowski I tried to cat the file and It would not work, it treated my targets as a command, I will come up with some good example if I have some more time. Thank you for your contribution, It was very helpful. – dwt.bar Dec 02 '20 at 17:03

GNU parallel bash

1 Answers1