How to loop a command through a file column values?

Question

I have a simple command like this

 grep 'X' results.dat | awk '{print $NF}'  > Y.dat

I want to loop this command taking the Xs from column 1 and the corresponding Ys from column 2 of the same file eg. NAMES

NAMES file has the format

C11-C12     p01
C13-C14-C17 P02
etc ..

so the first two steps in the loop should be like this

grep 'C11-C12' results.dat | awk '{print $NF}'  > p01.dat
grep 'C13-C14-C17' results.dat | awk '{print $NF}'  > p02.dat

Answer 1

A solution that doesn’t require looping in the shell:

awk 'pass==1 {  Xpatt[NR] = $1; Yfile[NR] = $2 ".dat"; printf "" > Yfile[NR] }
     pass==2 {
                for (i in Xpatt) {
                        if ($0 ~ Xpatt[i]) print $NF > Yfile[i]
                }
             }' pass=1 NAMES pass=2 results.dat

• First of all, awk allows you to specify variable assignments as command-line arguments, after the program, mixed in with the filenames, and not using -v.  They are executed at the point in the processing sequence that their position in the command line would suggest. So, in the above command,

  1. pass gets set to 1,
  2. the NAMES file is processed,
  3. pass gets set to 2, and then
  4. the results.dat file is processed.

  I guess I could have set pass=1 with a -v or in a BEGIN block.

  I use the pass variable to tell which file I’m reading.  This is commonly done by comparing NR to FNR, but that can lead to false indications if a file is empty.

  (Strictly speaking, I suppose that this script should check whether either of the files is empty, because, in that case, there’s no work to be done.)

• While pass==1 (we’re reading the NAMES file), save the X and Y values (pattern and filename) from columns 1 and 2 ($1 and $2) of that file.  Create the output file (Yfile[NR]) because, if we don’t do it here, we will not get (empty) output files for patterns that are not present in the results.dat file.  (If that’s OK with you, leave out the printf statement.)
• While pass==2 (we’re reading the results.dat file), loop through the patterns in the NAMES file and print the last word from every line that matches the pattern into the corresponding file — i.e., the equivalent of the OP’s grep X … | awk '{print $NF}' > Y.dat command.

Answer 2

Bash solution:

while read X Y remainder || [[ -n ${Y} ]]; do
    awk -v X="$X" '$0 ~ $X {print $NF}' results.dat > "$Y".dat
done < NAMES

• Generally, while IFS="q" read X Y remainder; do ...; done < NAMES will iterate over lines from NAMES. It will separate the values in each line based on the value of IFS (internal field separator). In this example, IFS is set to the letter q. IFS defaults to whitespace (space characters, tabs, or newlines). The first field is assigned the variable X, the second to Y, and the rest of the line to remainder.

  See also: Read columns from file into separate variables (Unix.SE).

  In the solution above, IFS is not specified because I assume your fields are already whitespace-separated.

  Note: if the fields in your NAMES file contains backslashes, then you need to use read -r to prevent read from interpreting backslashes as escape sequences.

• The ... remainder || [[ -n ${remainder} ]] part handles two things: any extra fields, if any, are stored in remainder; and handles the case if the last line of your input file doesn't end with newline \n (read returns a non-zero exit code when it encounters EOF).

  See also: Read a file line by line assigning the value to a variable (SO).

• Eliminate grep altogether: awk -v X="$X" '$0 ~ $X {print $NF}' results.dat > "$Y".dat. The -v option to awk defines a variable that can be used in the awk script.