How to remove empty lines at the end of a file using awk?

Question

I would like to remove all empty lines that are ONLY located at the end of the file using awk

I was able to successfully find a way to delete all of the empty lines at the top only with the following command:

awk '/^$/ && a!=1 {a=0} !/^$/ {a=1} a==1 {print}' file.txt

However, I didn't know how to reverse it so I could remove the bottom lines instead. I know I could just use the command above and pipe it with tac, but I prefer a direct approach using awk command only (if possible).

To clarify, a line is considered "empty" if it is "visually empty", i.e. contains at most spaces and/or tabs.

Answer 1

Awk

Since Awk reads the file sequentially, from the first to the last line, without external help (e.g. Tac) it can only figure whether a block of empty lines is at the end of the file when it actually reaches the end of the file.

What you can do is keep a variable with the empty lines (i.e., only newline characters, the default record separator RS) and print those empty lines whenever you reach a non-empty line:

awk '/^$/{n=n RS}; /./{printf "%s",n; n=""; print}' file

I don't understand why there is a difference between print n and printf n.

print appends the output record separator (ORS, by default a newline) to the expression to be printed. Thus you would get an extra newline if you tried it. You could also write it with a single output statement as in

awk '/^$/{n=n RS}; /./{printf "%s%s%s",n,$0,RS; n=""}' file

---

Ed or Ex

To print the output (just as Awk did), choose either of

printf '%s\n' 'a' '' '.' '?.?+1,$d' ',p' 'Q'  | ed -s file
printf '%s\n' 'a' '' '.' '?.?+1,$d' '%p' 'q!' | ex -s file

To directly apply the changes to the file, choose either of

printf '%s\n' 'a' '' '.' '?.?+1,$d' 'w' 'q'   | ed -s file
printf '%s\n' 'a' '' '.' '?.?+1,$d' 'x'       | ex -s file

To understand what's going on.

Command substitution

Shells strip trailing newline characters in command substitution.

printf '%s\n' "$(cat file)"

Mind that some shells will not handle large files and error with "argument list too long".

Inspired by this answer.

Answer 2

awk 'length == 0 { ++n; next } { for (i = 1; i <= n; ++i) print ""; n = 0 }; 1' file

or shortened, as suggested in comments,

awk 'length == 0 { ++n; next } { while (n) { print ""; --n } }; 1'

This keeps track of runs of empty lines in the counter n.

Whenever an empty line is seen (length == 0), the counter is incremented but nothing is output.

When a non-empty line is seen, the appropriate number of empty lines is first output before the current line. The counter n is also reset.

This avoids outputting empty lines from the end of the file.

---

Using standard sed:

sed -n -e :again -e N -e '/[^\n]/!b again' -e p file

This introduces an explicit loop that adds lines to the buffer until there is something other than just newlines in it. At that point, the buffer is output. If the input file ends while reading with N, the data in the buffer (which will be only newlines) will not be output.

Annotated code (the initial #n turns off the default output, just like using -n would do):

#n

# Label to branch to later.
:again

# Append next line of input to buffer
# with a delimiting newline.
N

# Branch (jump) to :again if there's
# only newlines in the buffer.
/[^\n]/!b again

# Output buffer.
p

Answer 3

This 1-pass approach will work whether the input is coming from a pipe or a file but has to store each block of empty lines in memory (which probably won't really be an issue unless you have billions of contiguous empty lines in your input):

awk 'NF{print s $0; s=""; next} {s=s ORS}' file

This 2-pass approach won't work if the input is a pipe, but will if the input is a file as you said in the question and uses almost no memory:

awk 'NR==FNR{if (NF) n=NR; next} FNR>n{exit} 1' file file

The above assumes that a line that contains only spaces is considered "empty". If that's wrong then change NF to /./.

Answer 4

Using awk and its range operator ,

awk '
!NF,NF{
  t = ($0 = t $0) ORS
  if (!NF) next
  t=""
}1' file

We accumulate the empty lines + one nonempty line in t. If a nonempty is seen we flush tHe buffer t and restart the process of accumulation.

---

With all-POSIX constructs, we can do it using the sed utility as shown:

sed -e '
  /^\n*$/!b
  $d;N;s/./&&/;D
' file

Answer 5

co=`awk '!/^$/{x=NR}END{print x}' filename`
co=$(($co+1))
j="$co,$"
sed -i "${j}d" filename

Tested and worked fine

Answer 6

Same idea as in this answer

What you can do is keep a variable with the empty lines (i.e., only newline characters, the default record separator RS) and print those empty lines whenever you reach a non-empty line

but using sed

sed -n -e '/^$/H' -e '/^$/!{x;/^$/!{s/\n//;p;z};x;p}'