2

QUESTION

In bash you can pass data from step to step by piping data:

program1 | program2 | program3 | ...

Or you can do this with variables (depending how each program work):

RES1=$(program1)
RES2=$(program2 $RES1)
...

I am looking for counterpart of such passing data in parallel.

EXAMPLE

I try to convert my "linear" script to the amazing parallel (this tool still do wonders for me :-D), but I have a problem with reading and using the output from the first step. Original version:

for fn in $(ls $REV *)
do
   DATA=$(sh script1.sh ${fn})
   sh script2.sh $DATA
done

Here how I tried to do this with parallel:

ls $REV * | parallel -j+0 DATA=$(sh script1.sh {}) \; sh script2.sh $DATA

However with this code myscript gets some broken data on input. Version with removed reading the result and second step altogether, works:

ls $REV * | parallel -j+0 sh script1.sh {}

So how to read the output from the first step of parallel and use it in next steps?

PROBLEM

For easier debugging let's say my first script (script1.sh) is:

echo "RECEIVED THIS ${1}"

And the main script is:

ls * | parallel -j+0 RES="$(sh script1.sh {})"

(I skipped the script2 here for testing capturing the output of script1). Then the result of entire execution is this:

/bin/bash: THIS: command not found
/bin/bash: THIS: command not found
/bin/bash: THIS: command not found
...
Community
  • 1
greenoldman
  • 6,086
  • 16
  • 54
  • 65
  • 1
    What do you mean by “some broken data”? Give examples that would allow us to reproduce the problem. Note that since you didn't quote variable and command substitutions, they're split on whitespace, which may be what breaks things — see http://unix.stackexchange.com/questions/131766/why-does-my-shell-script-choke-on-whitespace-or-other-special-characters – Gilles 'SO- stop being evil' Jul 26 '15 at 21:13
  • @Gilles, thank you for the hint, I totally updated the question and added the result of running the command. – greenoldman Jul 27 '15 at 05:51
  • By “first script”, do you mean `script1.sh`? What about `script2.sh`? And what command did you run to produce this output? – Gilles 'SO- stop being evil' Jul 27 '15 at 07:32
  • @Gilles, I skipped `script2` to test (it can be empty file) because the problem is with `script1`. I cannot read anything from `script1`, because when I run it with capture (i.e. `OUTPUT=$(sh script1)`) I get the error as shown in my post. In other words I can run `script1` in non-capture mode, but not in capture mode. Because of this problem I didn't test further steps, because it didn't make sense. – greenoldman Jul 27 '15 at 07:59
  • @don_crissti, typo, it should be `RES=....`. How do you merge 4 steps using first approach? One script solves the problem, but in such way, that I had to have plethora of scripts, just for connecting one program to another. This can be used for single case, but not as general solution. As I wrote, I am looking for counterpart of bash piping and/or capturing the output. – greenoldman Jul 27 '15 at 09:17
  • 1
    Use a function: myfunc() { DATA=$(sh script1.sh $1); sh script2.sh $DATA }; export -f myfunc; ls $REV * | parallel myfunc – Ole Tange Jul 28 '15 at 22:06
  • @OleTange, HA, beautiful. Thank you very much. Could you please post your comment as regular answer -- it is not possible to mark a comment as a solution. (Gee, CO fans stroke again -- I suspect it is not possible to add answers to the question anymore). – greenoldman Jul 29 '15 at 06:15

0 Answers0