8

Is there a canonical way to get all the logs from journalctl since a service was last restarted? What I want to do is restart a service and immediately see all the logs since I initiated the restart.

I came up with:

$ unit=prometheus
$ sudo systemctl restart $unit
$ since=$(systemctl show $unit | grep StateChangeTimestamp= | awk -F= '{print $2}')
$ sudo systemctl status -n0 $unit && sudo journalctl -f -u $unit -S "$since"

This will probably work, but I was wondering if there is a more concrete way to say: restart and give me all logs from that point onwards.

sdht0
  • 183
  • 4

2 Answers2

9

You can use the invocation id, which is a unique identifier for a specific run of a service unit.

It was introduced in systemd v232, so you need at least that version of systemd for this to work.

To get the invocation id of the current run of the service:

$ unit=prometheus
$ systemctl show -p InvocationID --value "$unit"
0e486642eb5b4caeaa5ed1c56010d5cf

And then to search journal entries with that invocation id attached to them:

$ journalctl INVOCATION_ID=0e486642eb5b4caeaa5ed1c56010d5cf + _SYSTEMD_INVOCATION_ID=0e486642eb5b4caeaa5ed1c56010d5cf

I found that you need both INVOCATION_ID and _SYSTEMD_INVOCATION_ID to get all the logs. The latter is added by systemd for logs output by the unit itself (e.g. the stdout of the process running in that service), while the former is attached to events taken by systemd (e.g. "Starting" and "Started" messages for that unit.)

Note that you don't need to filter by the unit name as well. Since the invocation id is unique, filtering by the id itself is sufficient to only include logs for the service you're interested in.

filbranden
  • 21,113
  • 3
  • 58
  • 84
  • 1
    Thanks @filbranden! This look good. – sdht0 Mar 17 '19 at 23:05
  • Although, the time based method includes the shutting down logs from the previous process, such as "Received SIGTERM, exiting gracefully...", which I think is also useful. So a way to track the exact time when systemd sent the sigterm signal would be nice. "StateChangeTimestamp=" captures that, right? – sdht0 Mar 17 '19 at 23:10
  • 1
    Or, you might need USER_INVOCATION_ID if you're running systemd services as user – karlicoss Jan 24 '20 at 21:29
0

While I learned something new from @filbranden's answer and accepted it, I found that the technique does not generalize. For example, if I want to stop a service and see its shutting down logs, that technique does not work because a stopped service does not have an invocationID.

I ended up using a more basic technique---simply storing the time before the commands and showing all logs since that time.

status_from() { sudo systemctl status -l --no-pager -n0 $1; echo; sudo journalctl -f -u $1 -S "$2"; }
start() { dt=$(date +'%a %Y-%m-%d %T %Z'); sudo systemctl start $1; status_from $1 "$dt"; }
stop() { dt=$(date +'%a %Y-%m-%d %T %Z'); sudo systemctl stop $1; status_from $1 "$dt"; }
restart() { dt=$(date +'%a %Y-%m-%d %T %Z'); sudo systemctl restart $1; status_from $1 "$dt"; }
sdht0
  • 183
  • 4