2

I want to remove the files which have only header and no data. I tried the below command and it is working fine, Only issue is it is giving exit status as 1 which should be zero.

wc -l /Path/File_*  | while read CNT FN; do [ $CNT -lt 2 ] && [  "$FN" != total ] && rm "$FN"; done

I am running this command at Datastage ETL tool and my job is failing.

is there any way to modify the command to get the exit status as 0

Aditya Kutcharlapati
  • 31
  • 4
  • I think this will help: https://unix.stackexchange.com/q/14270/388654 – Quasímodo Apr 28 '20 at 19:15
  • When `read` can not read more because there is no more data to read, it will fail. Is that what you mean is an issue? – Kusalananda Apr 28 '20 at 19:30
  • there are multiple files in the folder, but i want to remove the files which have only header and no data. For that, I have written to do wc and check if the count is less than 2 remove the file – Aditya Kutcharlapati Apr 28 '20 at 20:07

2 Answers2

2

To avoid having to read the whole files (like wc does), when you know after the first line that a file has at least 2 lines, I'd do (on a GNU system):

LC_ALL=C gawk -v ORS='\0' '
  FNR == 2 {nextfile}
  ENDFILE  {if (FNR < 2) print FILENAME}' /path/File_* |
  xargs -r0 rm -f

That's also more efficient in that it minimises the number of commands being run.

More reliable as it works with arbitrary file names.

As a functional difference with wc-based solutions: it would not delete files that contain one delimited line followed by one non-delimited one.

That one only returns a non-zero exit status if a file could not be removed (and was there in the first place).

Your problem is that the exit status of that pipe line is the exit status of the right-most command in it (as long as you don't use the pipefail option).

The right-most command here is the while loop. The exit status of a loop is that of the last command run in the body of the loop. In your case, it will be the [ "$FN" != total ] command run on the last line of the input, which will be non-zero unless there is only one /path/File_* file (in which case wc doesn't print the total).

If you changed it to:

[ "$CNT" -gt 1 ] || [ "$FN" = total ] || rm -f -- "$FN"

You'd only get a non-zero exit status if the last header file could not be removed.

Stéphane Chazelas
  • 522,931
  • 91
  • 1,010
  • 1,501
0

Your conditions return false at the end of the loop:

# false           # false              = false
[ $CNT -lt 2 ] && [  "$FN" != total ]

For example, if wc -l /path returns

2 total

and you run the command

                                 # true           # true              = true
wc -l *  | while read CNT FN; do [ $CNT = 2 ] && [  "$FN" = total ] && echo $CNT "$FN"; done
echo $?
0

but if you run

                                 # false           # true              = false
wc -l *  | while read CNT FN; do [ $CNT = 3 ] && [  "$FN" = total ] && echo $CNT "$FN"; done
echo $?  
1

In other words:

#false         # false      = false
[ 2 -eq 3 ] && [ bar = foo ]
echo $?
1

#true         # false      = false
[ 3 -eq 3 ] && [ bar = foo ]
echo $?
1

# false         # true      = false
[ 2 -eq 3 ] && [ foo = foo ]
echo $?
1

# true         # true      = true
[ 3 -eq 3 ] && [ foo = foo ]
echo $?
0

I would go with a for loop instead of a while loop and a function to return a value if no matches found:

foo() {
  ret=1 # no matches found
  for file in *.txt; do # your extension
    [[ $(wc -l "$file" | cut -d' ' -f1) -eq 1 ]] && echo "rm $file" && ret=0
  done
  return $ret
}
schrodingerscatcuriosity
  • 12,087
  • 3
  • 29
  • 57