7

I would like to store the default journal directory /var/log/journal/ on a mounted device, but I am not sure if I can do it due to the fact that journal is an important service which may already run before any device is mounted which would lead to the fact that suddenly the directly changes.

Is it safe/possible to store the journal directory on a mounted device and if yes, how to do it, if a simple mounting doesn't work.

Directory listing from the journal entries:

2019-03-18 22:16:41 root@AAEB-APP206LY:/var/log/journal/d41cf15550e34487abe7103b61fbf794 => ll
total 792M
drwxr-sr-x  1 root systemd-journal  884 Mar 12 06:35 ./
drwxr-sr-x  1 root systemd-journal   64 Feb 26 18:17 ../
-rw-r-----  1 root systemd-journal  96M Mar 18 22:16 system.journal
-rw-r-----  1 root root            120M Feb 26 18:17 system@d5301574c947425cb992f7839ae52cdb-0000000000000001-0005827c7effc14d.journal
-rw-r-----  1 root systemd-journal  96M Mar  5 12:29 system@d5301574c947425cb992f7839ae52cdb-0000000000051acb-000582cf3a7ba719.journal
-rw-r-----  1 root systemd-journal  96M Mar 12 06:35 system@d5301574c947425cb992f7839ae52cdb-00000000000872b4-000583572e31154d.journal
-rw-r-----+ 1 root systemd-journal 128M Mar 18 22:16 user-5000.journal
-rw-r-----+ 1 root root            128M Mar  5 12:29 user-5000@cf6acecdf28e48c790173a36447ec2e7-0000000000051ad9-000582cf3d435013.journal
-rw-r-----+ 1 root systemd-journal 128M Mar 12 06:35 user-5000@cf6acecdf28e48c790173a36447ec2e7-00000000000872b9-000583572e312040.journal

As you can see it is occupying almost 800 MB and this fills up the main partition. Therefore the idea to store it on a different filesystem.

Peter VARGA
  • 992
  • 3
  • 11
  • 28
  • You do not mean "on a mounted device". After all, all such filesystems are mounted, even the `/` filesystem. You mean "on some filesystem other than my `/` filesystem". – JdeBP Mar 18 '19 at 15:43
  • @JdeBP Yes, reading your comment I realize I described it wrong. You are right. I mean this. Does it _change_ anything and you could answer my question? – Peter VARGA Mar 18 '19 at 18:22
  • Are journal stored in inodes not in folders? I am a bit confused. Is this ext4? – Luciano Andress Martini Mar 18 '19 at 20:35
  • Right. That is not related to file system journal (as far as I know) but only to systemd init system. Is that still ok for you? – Luciano Andress Martini Mar 19 '19 at 12:07

2 Answers2

15

As far as I can tell, there is no way to change the location of systemd's predefined logging directories, /run/log/journal and /var/log/journal. It's possible via the Storage configuration option to choose which of those two options are used. But you can't change the path to /anotherfs/log/journal.

What you can do is make /var/log/journal a symlink to another directory, and have that directory be in another filesystem. Systemd-tmpfiles can automatically make this link.

Drop a file with this into /etc/tmpfiles.d/:

L     /var/log/journal     -    -    -     -      /anotherfs/journal

Now if /var/log/journal doesn't exist it will be a link to another filesystem and the journal will go there.

But then you will most likely run into another problem. Journald will switch from using a non-persistent journal in /run to the persistent journal in /var during boot, and flush the journal data from /run into /var. Obviously we need this switch to happen after /anotherfs is mounted! If the symlink pointed into the rootfs, or into the same fs as /var/log is on, then this wouldn't be issue and it would work fine as a way to change the path the journal data is at.

The way this ordering, mount filesystem then flush journal, is normally achieved is through systemd-journal-flush.service having a RequiresMountsFor=/var/log/journal property. If /var/log/journal is a symlink, systemd only waits for the filesystem the link itself is in to be mounted, not what the link points too. And so the ordering doesn't work.

We can get around this via bind mounts. Instead of symlinking /var/log/journal to another fs, we bind mount a directory from another fs (or directly mount the other fs, if the other fs is only for the journal) onto /var/log/journal.

Create a unit file which must be named var-log-journal.mount:

[Unit]
Description=Persistent Journal Storage Bind

[Mount]
What=/anotherfs/journal
Where=/var/log/journal
Type=none
Options=bind

[Install]
WantedBy=local-fs.target

Install and enable this. Now the journal flush unit will see there is a mount unit for var-log-journal (it only looks for that exact name!) and wait for it. The mount unit will bind mount the directory /anotherfs/journal onto /var/log/journal. Systemd will automatically add ordering dependencies to wait for /anotherfs to be mounted if it needs that and will automatically create the mountpoint /var/log/journal if it's not already there.

TrentP
  • 439
  • 4
  • 7
  • I was having an issue with .journal~ files, from unclean shutdowns, filling up my journal partition. They were never being cleaned up despite my journald.conf file saying they should. My journal location was being symlinked to another partition. Switching to this specific bind .mount fixed the issue. – Matt Minga Mar 28 '23 at 20:49
-2

From the systemd-journald manual page:

The journal service stores log data either persistently below /var/log/journal or in a volatile way below /run/log/journal/ (in the latter case it is lost at reboot). By default, log data is stored persistently if /var/log/journal/ exists during boot, with an implicit fallback to volatile storage otherwise. Use Storage= in journald.conf(5) to configure where log data is placed, independently of the existence of /var/log/journal/.

So it is possible and safe to store the journal at some other place. You will just have to add/change

Storage=/some-filesystem-with-free-space/some-dir/

in one of the journald configuration files.

Uwe Ohse
  • 142
  • 3
  • That doesn't work. Take a look at how Storage works, `Controls where to store journal data. One of "volatile", "persistent", "auto" and "none".`. It's for controlling how systemd stores the data, not the exact file path. – TrentP Mar 05 '20 at 01:17