27

I just noticed this verify option for journalctl and decided to give it a go, it's showing corruption, what might cause that? and what if anything should I do about it? should I investigate further?

journalctl --verify
PASS: /var/log/journal/19184893a1d645c7a43729e79b10a876/user-1000.journal
Invalid object contents at 3733856░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░   0%
File corruption detected at /var/log/journal/19184893a1d645c7a43729e79b10a876/system.journal:3733856 (of 91734016, 4%).
FAIL: /var/log/journal/19184893a1d645c7a43729e79b10a876/system.journal (Bad message)
Invalid object contents at 21575496░░░░░░░░░░░░░░░░░░░░░░░░░░░░░  45%
File corruption detected at /var/log/journal/19184893a1d645c7a43729e79b10a876/system@60e058db556e4de4b256d0b1ff176aa4-0000000000000001-0004e0b436d20aa1.journal:21575496 (of 44052480, 48%).
FAIL: /var/log/journal/19184893a1d645c7a43729e79b10a876/system@60e058db556e4de4b256d0b1ff176aa4-0000000000000001-0004e0b436d20aa1.journal (Bad message)
PASS: /var/log/journal/19184893a1d645c7a43729e79b10a876/user-1000@60e058db556e4de4b256d0b1ff176aa4-0000000000000a91-0004e0b4ff9a949a.journal
PASS: /var/log/journal/19184893a1d645c7a43729e79b10a876/user-1001.journal
sourcejedi
  • 48,311
  • 17
  • 143
  • 296
xenoterracide
  • 57,918
  • 74
  • 184
  • 250

3 Answers3

22

Currently, journalctl can detect corrupt logs but has no "fsck" type command to attempt repair. The journald will automatically switch to writing a new "clean" file as soon as it detects the problem, so theoretically data loss should be minimal.

Until there is a file-repairing command, finding the corrupt journal file and removing it is the only cure. You can find more on this in our Fedora mega-thread over making journal-only logging the default:

For tail corruptions your normal journalctl tool will provide you with as much information as is possible to salvage from the file. It will output the last complete log line and then finish. This is pretty close to how good you can get.

Things are different for corruptions in the middle. We have no nice tool for salvaging data from such corruption, but they could be written relatively easily. However, since they are highly unlikely due to the "append-only" model of the journal this hasn't been on our TODO list.

Of course, if you can identify what caused the problem initially and report it, that would be nice.

Community
  • 1
mattdm
  • 39,535
  • 18
  • 99
  • 133
  • what is an FAA? – xenoterracide Aug 23 '13 at 04:40
  • @xenoterracide some sort of weird phone typo is what that is. :) – mattdm Aug 23 '13 at 12:09
  • You wrote this on your phone? Impressive 8-). Thanks for the typo, I tried googling for FAA this morning and wasted like 30 minutes thinking it was something ;-). – slm Aug 23 '13 at 14:18
  • 1
    @slm Yeah, there's a long bus ride from the office. :) Sorry for the time wasting -- hope you at least learned something about aviation. – mattdm Aug 23 '13 at 14:44
  • @mattdm - LOL, thanks most certainly did not learn anything useful wrt FAA 8-). – slm Aug 23 '13 at 14:51
  • 1
    on my system at least corrupt journals seem to be common... is there any advice about how to identify what caused the problem? – xenoterracide Aug 23 '13 at 21:43
  • 2
    On every systemd system I have seen, corrupt journal logs are common. This answer is now almost 5 years old and as far as I can tell, there is no solution for this issue. – MountainX Mar 26 '18 at 19:37
3

It seems to be related to this thread on the ArchLinux wiki, titled: journalctl issues. It seems to have something to do with this setting SystemMaxUse in /etc/systemd/journald.conf.

The thread isn't conclusive but several people were having some luck in either clearing out the logs under /var/log/journal/* or increasing the value for SystemMaxUse.

slm
  • 363,520
  • 117
  • 767
  • 871
  • unfortunately playing with that value doesn't seem to have helped. I'm suspect clearing logs helps because it deletes the invalid logs. – xenoterracide Aug 23 '13 at 00:14
  • @xenoterracide - I'll keep looking. Yeah cleaning out the logs seemed nonsensical to me too, but people were reporting it so I figured it was worth a shot. – slm Aug 23 '13 at 01:10
0
# journalctl --flush
# journalctl --verify
Jeff Schaller
  • 66,199
  • 35
  • 114
  • 250
Alex
  • 356
  • 3
  • 10
  • What does the `journalctl --flush` command do ? – SebMa Aug 02 '23 at 09:13
  • i just quote from man journalctl : (--flush) == >Asks the journal daemon to flush any log data stored in /run/log/journal into /var/log/journal, if persistent storage is enabled. This call does not return until the operation is complete. Note that this call is idempotent: the data is only flushed from /run/log/journal into /var/log/journal once during system runtime, and this command exits cleanly without executing any operation if this has already happened. This command effectively guarantees that all data is flushed to /var/log/journal at the time it returns. – Alex Aug 04 '23 at 12:01