Loop through the lines of two files in parallel

Question

The object of the script I'm making is to compare two series of files. The file names are themselves stored into two separate files, one path per line. My idea is to have two while read loops, one for each list of file names, but how can I mix the two loops together?

while read compareFile <&3; do     
 if [[ ! $server =~ [^[:space:]] ]] ; then  #empty line exception
    continue
 fi   
    echo "Comparing file - $compareFile"
 if diff "$compareFile" _(other file from loop?_) >/dev/null ; then
     echo Same
 else
      echo Different
 fi 
done 3</infanass/dev/admin/filestoCompare.txt

I need to be able to compare files from two different lists at the same time through two while read loops... Is this even possible?

Answer 1

You don't need two loops; you just need to read from two files in the one loop.

while read compareFile1 <&3 && read compareFile2 <&4; do     
 if [[ ! $server =~ [^[:space:]] ]] ; then  #empty line exception
    continue
 fi   
    echo "Comparing file - $compareFile"
 if diff "$compareFile1" "$compareFile2" >/dev/null ; then
     echo Same
 else
      echo Different
 fi 
done 3</infanass/dev/admin/filestoCompare.txt 4<other_file

Answer 2

Method 1: use what you know

Since you already know how to loop over one file, you could combine the files and then process the combined files. The command paste joins two files line by line. It puts a tab between the lines coming from the two files, so this solution assumes that there are no tabs in your file names. (You can change the separator but you have to find a character that isn't present in a file name.)

paste -- "$list1.txt" "list2.txt" |
while IFS=$'\t' read -r file1 file2 rest; do
  diff -q -- "$file1" "$file2"
  case $? in
    0) status='same';;
    1) status='different';;
    *) status='ERROR';;
  esac
  echo "$status $file1 $file2"
done

If you want to skip blank lines, you need to do it in each file separately, since paste might match a blank line from one file with a non-blank line from another file. You can use grep to filter the non-blank lines.

paste -- <(grep '[^[:space:]]' "$list1.txt") <(grep '[^[:space:]]' "list2.txt") |
while IFS=$'\t' read -r file1 file2 rest; do
  …

Note that if the two files have different lengths, you'll get an empty $file2 (regardless of which list ended first).

Method 2: loop over two files

You can put as complex a command as you like in the condition of the while loop. If you put read file1 <&3 && read file2 <&4 then the loop will run as long as both files have a line to read, i.e. until one file runs out.

while read -u 3 -r file1 && read -u 4 -r file2; do
  …
done 3<list1..txt 4<list2.txt

If you want to skip over blank lines, it's a little more complicated, because you have to do the skipping in the two files independently. The easy way is to break the problem into two parts: skip the blank lines from one file, and process the non-blank lines. One method to skip the blank lines is to process through grep as above. Watch out for the necessary space between the < redirection operator and the <( that starts a command susbtitution.

while read -u 3 -r file1 && read -u 4 -r file2; do
  …
done 3< <(grep '[^[:space:]]' "$list1.txt") 4< <(grep '[^[:space:]]' "list2.txt")

Another method is to write a function that behaves like read but skips blank lines. This function can work by calling read in a loop. It doesn't have to be a function, but a function is the best approach, both to organize your code and because that piece of code needs to be called twice. In the function, ${!#} is an instance of the bash construct ${!VARIABLE} which evaluates to the value of the variable whose name is the value of VARIABLE; here the variable is the special variable # which contains the number of positional parameter, so ${!#} is the last positional parameter.

function read_nonblank {
  while read "$@" &&
        [[ ${!#} !~ [^[:space:]] ]]
  do :; done
}
while read_nonblank -u 3 -r file1 && read_nonblank -u 4 -r file2; do
  …
done 3<list1..txt 4<list2.txt

Answer 3

One approach would be to use read -ra instead of just read. Assuming the filestoCompare.txt contained 2 columns with the file names in each, the read -ra would read both columns in at the same time and assign them into an array, compareFile. This array could then be accessed so that index 0 would be the first file and index 1 would be the 2nd file each time through the while loop.

Example

Say I have this file: filestoCompare.txt, and it contains the following:

file1 file2
file3 file4
file5 file6

The command to go through this file would be as follows:

$ while read -ra a ; do printf "%s\t%s\n" ${a[0]} ${a[1]}; done < filestoCompare.txt
file1   file2
file3   file4
file5   file6

If the 2 files are indeed separate files such as:

#list1
file1
file2
file3

#list2
file4
file5
file6

They can be joined together with the paste command like so:

$ paste list1 list2 > list1and2

Here's the contents of list1and2:

$ cat list1and2
file1   file4
file2   file5
file3   file6