20

I have a bash script that runs as long as the Linux machine is powered on. I start it as shown below:

( /mnt/apps/start.sh 2>&1 | tee /tmp/nginx/debug_log.log ) &

After it lauches, I can see the tee command in my ps output as shown below:

$ ps | grep tee
  418 root       0:02 tee /tmp/nginx/debug_log.log
3557 root       0:00 grep tee

I have a function that monitors the size of the log that tee produces and kills the tee command when the log reaches a certain size:

monitor_debug_log_size() {
                ## Monitor the file size of the debug log to make sure it does not get too big
                while true; do
                                cecho r "CHECKING DEBUG LOG SIZE... "
                                debugLogSizeBytes=$(stat -c%s "/tmp/nginx/debug_log.log")
                                cecho r "DEBUG LOG SIZE: $debugLogSizeBytes"
                                if [ $((debugLogSizeBytes)) -gt 100000 ]; then
                                                cecho r "DEBUG LOG HAS GROWN TO LARGE... "
                                                sleep 3
                                                #rm -rf /tmp/nginx/debug_log.log 1>/dev/null 2>/dev/null
                                                kill -9 `pgrep -f tee`
                                fi
                                sleep 30
                done
}

To my surprise, killing the tee command also kills by start.sh instance. Why is this? How can I end the tee command but have my start.sh continue to run? Thanks.

Jeff Schaller
  • 66,199
  • 35
  • 114
  • 250
PhilBot
  • 69
  • 8
  • 22

2 Answers2

36

When tee terminates, the command feeding it will continue to run, until it attempts to write more output. Then it will get a SIGPIPE (13 on most systems) for trying to write to a pipe with no readers.

If you modify your script to trap SIGPIPE and take some appropriate action (like, stop writing output), then you should be able to have it continue after tee is terminated.

Better yet, rather than killing tee at all, use logrotate with the copytruncate option for simplicity.

To quote logrotate(8):

copytruncate

Truncate the original log file in place after creating a copy, instead of moving the old log file and optionally creating a new one. It can be used when some program cannot be told to close its logfile and thus might continue writing (appending) to the previous log file forever. Note that there is a very small time slice between copying the file and truncating it, so some logging data might be lost. When this option is used, the create option will have no effect, as the old log file stays in place.

David Foerster
  • 1,505
  • 1
  • 11
  • 18
Wildcard
  • 35,316
  • 26
  • 130
  • 258
  • 9
    You'd also want to use `tee -a` for `tee` to open the file in append mode, otherwise, tee will carry on writing into the file at the same offset after you truncate it (and on systems that don't support sparse files like on macOS that will reallocate the section of the file leading up to that position, taking twice as much disk space). – Stéphane Chazelas Jan 05 '18 at 08:31
  • 5
    Other option would be to pipe to `logger -s` for syslog to take care of the logging (`-s` to also print on stderr). – Stéphane Chazelas Jan 05 '18 at 08:34
  • 1
    +1 for `logrotate`. Great program – anna328p Jan 05 '18 at 20:41
  • 2
    Or on a system using systemd and journald, systemd-cat instead of logger. Then you get lots of output filtering and rotation for free. – Zan Lynx Jan 05 '18 at 21:16
3

Explaining The "Why"

In short: If writes failing didn't cause a program to exit (by default), we'd have a mess. Consider find . | head -n 10 -- you don't want find to keep running, scanning the rest of your hard drive, after head has already taken the 10 lines it needed and proceeded.

Doing It Better: Rotate Inside Your Logger

Consider the following, which doesn't use tee at all, as a demonstrative example:

#!/usr/bin/env bash

file=${1:-debug.log}                     # filename as 1st argument
max_size=${2:-100000}                    # max size as 2nd argument
size=$(stat --format=%s -- "$file") || exit  # Use GNU stat to retrieve size
exec >>"$file"                           # Open file for append

while IFS= read -r line; do              # read a line from stdin
  size=$(( size + ${#line} + 1 ))        # add line's length + 1 to our counter
  if (( size > max_size )); then         # and if it exceeds our maximum...
    mv -- "$file" "$file.old"            # ...rename the file away...
    exec >"$file"                        # ...and reopen a new file as stdout
    size=0                               # ...resetting our size counter
  fi
  printf '%s\n' "$line"                  # regardless, append to our current stdout
done

If run as:

/mnt/apps/start.sh 2>&1 | above-script /tmp/nginx/debug_log

...this will start out by appending to /tmp/nginx/debug_log, renaming the file to /tmp/nginx/debug_log.old when over 100KB of contents are present. Because the logger itself is doing the rotation, there's no broken pipe, no error, and no data loss window when rotation takes place -- every line will be written to one file or another.

Of course, implementing this in native bash is inefficient, but the above is an illustrative example. There are numerous programs available which will implement the above logic for you. Consider:

  • svlogd, the service logger from the Runit suite.
  • s6-log, an actively-maintained alternative from the skanet suite.
  • multilog from DJB Daemontools, the granddaddy of this family of process supervision and monitoring tooling.
Charles Duffy
  • 1,651
  • 15
  • 22