Modification with Script error

Question

I just write few lines to grep smallest value in my files and it is giving me correct result but repeating lines two times can you fix the bug

What I am doing:

• Greping all files
• Removing header
• sorting in scientific notation using column nine
• taking the first line that is the smallest after sort and printing using awk
• I want file name so printed $i too

Script:

#!/bin/bash

for i in `ls -v *.txt` 
do 
smallestPValue=`sed 1d $i | sort -k9 -g | head -1 | awk '{print $0}'` 

echo  $i  $smallestPValue >> smallesttPvalueAll.txt
done

output

U1.text 4 rsxxx 1672175 A ADD 759 0.0751 4.918 1.074e-06
U1.txt 4 rsxxxx 1672175 A ADD 759 0.0751 4.918 1.074e-06
U2.txt  16 rsxxxx 596342 T ADD 734 -0.05458 -5.204 2.535e-07
U2.txt 16 rsxxxx 596342 T ADD 734 -0.05458 -5.204 2.535e-07
U3.txt 2 rsxxxx 12426 T ADD 722 0.06825 5.285 1.669e-07

I am getting repetitions for few lines while some are just fine as U3 above is coming once and that's what I want. I can easily get rid of duplicated lines by uniq or sort -u but just curious what is causing this

Desired output each line repeated once

Answer 1

If I'm interpreting it right, you can probably do what you're trying to do with just awk and sort - no need for a loop, or parsing ls (subtle hint: DON'T DO THAT!), or head or sed.

awk 'FNR > 1 {print FILENAME, $0}' *.txt | sort -k10 -g | sort -u -k1,1

This skips the first line of each file, then prints all remaining lines prefixed with the filename and a space (awk's default output record separator or ORS). It then pipes it through sort to do a generic numeric sort on field 10. Finally, it does a unique sort of the first field only (-k1,1, the filename), so that only the first line with that filename is output.

Note that we have to sort on field 10 here, not field 9 because we've added the filename as the first field so all other field numbers are incremented by 1.

FNR and FILENAME are built-in awk variables. FNR is the line number ("input record number" in awk-lingo) of the current file, and FILENAME is the current filename.

---

here's another way of doing it, this time using only awk:

#!/usr/bin/awk -f

FNR > 1 && (! s[FILENAME] || $9 < s[FILENAME]) {
  s[FILENAME]=$9;
  l[FILENAME]=$0
};

END {
  for (f in s) {
    print f, l[f]
  }
}

save it as, e.g. smallest-pvalue.awk, make it executable with chmod +x smallest-pvalue.awk and run it as ./smallest-pvalue.awk *.txt.

This awk script keeps track of the smallest value seen for field 9 of each input file in an array called s, and also keeps the matching input line in array l.

Once it has processed all the files, it prints out the filename and the line containing the smallest 9th field for each file.