Can't Break the while read loop even if condition is met

Question

This should be simple but I am missing something, need some help. My requirement is to read the log file via tail to get latest logs, grep Download Config & Copying all files of and write it in MyOwnLogFile.log but I want this to stop as soon as .myworkisdone file appears in /usr/local/FOLDER/ One thing is sure that .myworkisdone will be generated at the last when all logs are done… but the script just continues to read the log file and never comes out of it, even if the file is created.

while [[ ! -e /usr/local/FOLDER/.myworkisdone ]];
do
    sudo tail -f -n0  /var/log/server22.log | while read line; do echo "$line" | grep -e 'Downloading Config’ -e ‘Copying all files of' ; done  >> /var/tmp/MyOwnLogFile.log
done

I also tried until instead of while to check the file but still the script cant break the spell of reading the log file. Thank you in advance.

Answer 1

As said by @ikkachu, tail -f is an infinite process. However you may use the inner loop to initiate the breaking:

while :
do
    tail -f server.log | while IFS= read -r line
    do
        printf '%s\n' "$line" | grep 'word' >> ownlog.log
        if [ -e killer.file ] ; then break ; fi
    done
    break 
done

• Why does break 2 in the if-loop not work?

The pipe initiates a subshell, the break command cannot exit this subshell. Hence another break in the outer loop.

• When does the script stop?

Once the killer.file is existing, the pending | while read ... still awaits another entry and holds open the whole thing. I.e. you need the next line, i.e. (with dummy files):

 echo test >> server.log
 #"normal" output - nothing happens

 echo word >> server.log
 #matching output - logged to ownlog.log
 
 touch killer.file
 #killer file appears - breaks cannot take effect
 #as "read" still awaits signal.

 echo test2 >> server.log
 #break takes effect

This is especially important if the killer.file would also indicate that there are no more logs written to server.log, so the process remains constantly open.

Answer 2

You need to separate the checking for the "stop file" from the reading of the log file, because tail -f will not exit. The following shell script does that by having a background task check for the existence of the "stop file" once a second. Whenever the file appears, it kills the tail -f process with a PIPE signal (the same signal it would have received if grep had stopped reading). When tail -f terminates due to the PIPE signal, the grep process terminates too since its input stream closes.

#!/bin/sh

stopfile=/usr/local/FOLDER/.myworkisdone

set -- tail -f /var/log/server22.log 

"$@" | {
        while true; do
                if [ -e "$stopfile" ]; then
                        pkill -PIPE -fx "$*"
                        break
                fi
                sleep 1
        done &

        grep -e 'Downloading Config' \
             -e 'Copying all files of'
} >/var/tmp/MyOwnLogFile.log

This sets the positional parameters to the tail -f command. This makes it a bit easier to refer to it with pkill later. The pkill utility is called with both -f and -x to be able to kill exactly the command that we started earlier.

The value "$*" is a single string consisting of all positional parameters (the tail -f command) with a space in-between them (by default).

Using bash and a named array for the tail -f command:

#!/bin/bash

stopfile=/usr/local/FOLDER/.myworkisdone

talicmd=( tail -f /var/log/server22.log )

"${tailcmd[@]}" | {
        while true; do
                if [ -e "$stopfile" ]; then
                        pkill -PIPE -fx "${tailcmd[*]}"
                        break
                fi
                sleep 1
        done &

        grep -e 'Downloading Config' \
             -e 'Copying all files of'
} >/var/tmp/MyOwnLogFile.log

(I'm also noticing that a couple of the single quotes around your regular expressions used with grep in the question are not ordinary single quotes but "fancy quotes".)

Answer 3

perl has a module called File::Tail designed to monitor log file(s), so your program can take actions if conditions are met (e.g. printing the line if it matches a pattern).

     #!/usr/bin/perl

     use File::Tail;
     tie *FH,"File::Tail",(name=>shift,nowait=>1);

     while (<FH>) {
         print if (m/Downloading Config|Copying all files of/);
         exit if ( -e '/usr/local/FOLDER/.myworkisdone')
     }

save as, e.g., watch.pl, make it executable with chmod +x watch.pl and run it as:

$ ./watch.pl /var/log/server22.log

or just embed it as a one-liner in a shell script, same as you would for an awk or sed script:

perl -MFile::Tail -e 'tie *FH,"File::Tail",(name=>shift,nowait=>1);
      while (<FH>) {
        print if (m/Downloading Config|Copying all files of/);
        exit if ( -e '/usr/local/FOLDER/.myworkisdone')
      }' /var/log/server22.log

This monitors the log file named as the first arg to the script (name=>shift). It prints any logfile lines that match the pattern, and exits if the .myworkisdone file exists.

nowait=>1 prevents the script from blocking while reading the input file. It's less than ideal, but OK for a quick-and-dirty script. A better solution would be to use the File::Tail's select() method. See man File::Tail.

BTW, the select() method also makes it easy to monitor multiple log files at once.