8

Towards the end of the "startup sequence"1, I see a longish series of diagnostic messages fly by very quickly, right before I see the login prompt2.

AFAICT, most, if not all, of the lines making up this short-lived output begin with either of the strings shown below

[  OK  ]
[FAILED]

...where the OK is in green, and the FAILED is in red3.

These messages flash too briefly for me to read.

My question is:

Is there a way to make these messages easier to read?


Possible solutions that come to mind include (in order of preference):

  1. teeing (or simply redirecting) those messages verbatim4 to some persistent log file;
  2. enabling a paging-type mechanism (Press any key to continue...);
  3. inserting a pause (of configurable length) after those messages are printed;
  4. enabling some key (or key combination) to pause the output to the screen5.

EDIT: based on the comments I've gotten so far, I must conclude that the word verbatim in (1) above is either not being understood, or not being taken seriously, even though I have emphasized as much as I can. I would make it flash if I could...


EDIT2: the suggestion that meuh gave in the comments seems promising to me, but I have not been able to get it to work yet. Here's what I did:

First, I added the following at the end of /etc/rsyslog.conf:

# Save boot messages also to boot.log
local7.* /var/log/boot.log

...and rebooted. I saw the usual diagnostic messages fly by, but no /var/log/boot.log file was created.

Then, in the (admittedly unlikely) event that the /var/log/boot.log has to exist already before rsyslog can write to it, I executed (as root):

touch /var/log/boot.log
chgrp adm /var/log/boot.log
chmod 640 /var/log/boot.log

...where the chgrp and chmod commands were intended to make the ownership and permissions of /var/log/boot.log match those of all the other log files under /var/log. Then I rebooted, saw the messages, etc. The file /var/log/boot.log remained empty after this reboot.

(I got the same non-result when I changed the permissions of /var/log/boot.log to 666.)

I grep'ed the output of journalctl --boot and the files under /var/log for anything I could think of that may point to something awry with my rsyslog, but did not find anything. (I'm not at all familiar with rsyslog, so I'm sure my search was pretty inept.)

It's clear that what I've done so far is not sufficient to enable the desired logging. I'm now looking for whatever is that I'm missing. I have not been able to find much relevant documentation, though. For example, neither rsyslog.conf(5) nor rsyslogd(8) deigns to explain what local7 is (rsyslog.conf(5) is at least gracious enough to mention it once, without giving any further information).


EDIT3

Distro information:

$ lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description:    Debian GNU/Linux 8.3 (jessie)
Release:    8.3
Codename:   jessie
$ uname -a
Linux myhost 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt20-1+deb8u3 (2016-01-17) x86_64 GNU/Linux

EDIT4

Additional potentially relevant information:

$ cat /lib/systemd/system/rsyslog.service
[Unit]
Description=System Logging Service
Requires=syslog.socket
Documentation=man:rsyslogd(8)
Documentation=http://www.rsyslog.com/doc/

[Service]
Type=notify
ExecStart=/usr/sbin/rsyslogd -n
StandardOutput=null
Restart=on-failure

[Install]
WantedBy=multi-user.target
Alias=syslog.service
$ cat /proc/$(pgrep rsyslogd)/limits
Limit                     Soft Limit           Hard Limit           Units     
Max cpu time              unlimited            unlimited            seconds   
Max file size             unlimited            unlimited            bytes     
Max data size             unlimited            unlimited            bytes     
Max stack size            8388608              unlimited            bytes     
Max core file size        0                    unlimited            bytes     
Max resident set          unlimited            unlimited            bytes     
Max processes             128529               128529               processes 
Max open files            1024                 4096                 files     
Max locked memory         65536                65536                bytes     
Max address space         unlimited            unlimited            bytes     
Max file locks            unlimited            unlimited            locks     
Max pending signals       128529               128529               signals   
Max msgqueue size         819200               819200               bytes     
Max nice priority         0                    0                    
Max realtime priority     0                    0                    
Max realtime timeout      unlimited            unlimited            us        

$ sudo ls /proc/$(pgrep rsyslogd)/fd | wc -l
10

1 I.e. that which takes place when I (re)boot my machine.

2 FWIW, multi-user.target is my default.

3 The remaining text is all in white over a black background. This is true of the subsequent login prompt.

4 I find completely unacceptable any solution that does not let me see the exact text of these messages as they appeared during the startup sequence. Since, invariably, I am not intimately familiar with whatever any of these diagnostic messages is referring to, it is not possible for me to recognize all the ways in which the underlying information conveyed by the original message may be paraphrased, spread over multiple other messages, subsumed in other messages, etc. (Only by searching online for the exact wording of the original message do I have any hope of finding a solution to the problem.) Everything I've tried so far, including journalctl -b and dmesg has failed at giving me the original messages verbatim. For example, when I run startup I can see only one red FAILED, but journalctl --boot | grep FAILED | wc -l returns 0, and journalctl --boot | grep -i FAILED | wc -l returns 1086. Neither of these is what I'm looking for.

5 In my system, I would have less than a second to press such key or key combination, and there is no forewarning of when this brief interval starts. Unless one is able to configure the duration of the interval during which such a keypress must happen, any key-press-based solutions is too impractical to be anything more than a last-resort maneuver. Also, FWIW, I did try pressing either the Scroll
Lock
or the Pause/
Break
key when the messages flashed, but neither made any difference.

kjo
  • 14,779
  • 25
  • 69
  • 109
  • 5
    Doesn't `journalctl -b` (as root) give you exactly that, i.e. _see the exact text of these messages as they appeared during the startup sequence_ ? – don_crissti Feb 21 '16 at 17:48
  • Have you try with dmesg? – Romeo Ninov Feb 21 '16 at 17:52
  • 2
    Depending on your system you may find the messages in file `/var/log/boot.log` – meuh Feb 21 '16 at 17:53
  • @don_crissti: no, that's what I took pains to explain in fn 4: it is possible that the output of `journalctl -b` contains information that, to the expert, would be for all intents and purposes equivalent to what is conveyed by the startup messages, but it does not contain those messages verbatim. I'll edit my post to make this more explicit. – kjo Feb 21 '16 at 17:55
  • Just wanted to chime in that i recently had to discover on a systemd/linux box: "hey what are all those new red "Failure to.." at start-up?! damn they're gone." All the old tricks "dmesg" and "more /var/log/messages" were no more, but exactly what don_crissti suggests above worked. This uncovered the messages (if not why the hell they've started up): journalctl --boot | grep -i fail – Theophrastus Feb 21 '16 at 17:56
  • 2
    @Theophrastus: I'm beginning to see why so many Linux users detest `systemd`, and I'm about to join their ranks myself... I have edited my fn 4 to spell out (even more) why `journalctl --boot | grep -i fail`, for example, is not what I'm looking for. – kjo Feb 21 '16 at 18:05
  • @don_crissti: with any of the versions of `journalctl` you mentioned, if I pipe the output through `grep OK` I get **zero** hits, even though I see tons of `OK`'s flashing right before the login prompt. This proves to me, beyond any doubt, that whatever it is that these commands print out, it is most definitely not what I am looking for. – kjo Feb 21 '16 at 18:09
  • @meuh: there's no `/var/log/boot.log` in my system. There's a `/var/log/boot` file, but its modification time is `Feb 13 13:39`. Since I've rebooted my system multiple times since then, and seen all those diagnostic messages at every one of those reboots, this file can't possibly contain what I'm looking for. There's also a file `/var/log/bootstrap.log`, but it's empty. – kjo Feb 21 '16 at 18:17
  • 1
    My `/var/log/boot.log` is created by an entry in `/etc/rsyslog.conf` with the line `local7.* /var/log/boot.log` (preceded by the comment *Save boot messages also to boot.log* in Fedora 22). It has all the ok's and failed's in colour (vt100 escape sequences). – meuh Feb 21 '16 at 18:21
  • @meuh: that looks like the right direction, but I have not gotten it to work yet. I added the lines you mentioned to `/etc/rsyslog.conf`, and rebooted, but the `/var/log/boot.log` file was not created. I then created the file (`touch /var/log/boot.log`), plus `chgrp adm`'d and `chown 640`'d it (to match the ownership/permissions of the other log files under `/var/log`), and rebooted. The `/var/log/boot.log` file remained empty. I searched the logs for any error relating to my modification of `/etc/rsyslog.conf`, but did not find anything. I have not yet found any documentation on this. – kjo Feb 21 '16 at 18:46
  • 1
    On the distro I'm running, (Mint, a variant of Ubuntu), I find these messages verbatim in /var/log/boot.log. What system are you running? That is, what is the output of `lsb_release -a`, or, failing that, `uname -a`? What's the actual problem you're trying to solve? The "general case", might be another way to put it. Is it simply to better understand what your system is doing? In that case, it may be rewarding to research the meanings of the logs you *can* find. At any rate, these messages are almost certainly logged *somewhere*. I'd suggest exploring your /var/log directory to start. – Jon Carter Feb 21 '16 at 19:18
  • @JonCarter: I've added an EDIT3 to my post to answer one of your questions. The short version is: Debian 8.3/jessie. – kjo Feb 21 '16 at 19:41
  • 3
    kjo, the only difference that I see between the messages during boot and those logged to the `journal` is the presence of `[OK]`/`[FAILED]`. The messages are otherwise identical. _The right way to diagnose/troubleshoot failed_ units is via `systemctl` though, just so you know. I don't know if you can pause the boot process via a keyboard shortcut (they say CTRL+S/CTRL+Q should work but it doesn't, at least not on i915/KMS). Still, you could [Disable clearing of boot messages](http://tinyurl.com/lhp3amo) and scroll through those messages on TTY1 with Shift+PgUp/Down. – don_crissti Feb 21 '16 at 19:52
  • as per your "EDIT3" wherein we discover that you're running Debian Jessie (aka "stable" currently), it should be noted that that represents a somewhat half-hearted/cautious implementation of systemd (which i can be sympathetic to). That is, you might want to search around about how you activate more systemd features (which you seem to be pursuing) specifically for Jessie (or switch to "testing"). For example, adding a kernel parameter of "debug", viz: https://www.debian.org/releases/stable/i386/release-notes/ch-information.html – Theophrastus Feb 21 '16 at 20:01
  • @don_crissti: I don't understand how the output of `journalctl -b [-a]` can come *even close* to resembling the diagnostic messages I see at the end of startup. Even in the numbers of lines, the two sets of information differ by a couple orders of magnitude (I see a couple dozen lines' worth of messages at the end of startup, they don't even fill my screen, whereas the output of `journalctl -b [-a]` shows > 2200 lines). I must conclude that our systems are configured radically differently. – kjo Feb 21 '16 at 20:08
  • 1
    No. That is quite normal. Everything logs to the journal, so you get those boot messages mixed with other stuff. Try like this: `journalctl /usr/lib/systemd/systemd -b` to "emulate" the stuff that you actually see during boot. – don_crissti Feb 21 '16 at 20:11
  • `systemctl --failed`... – jasonwryan Feb 21 '16 at 20:17
  • 2
    Maybe the following Q/A gives something: http://superuser.com/questions/480370/linux-boot-stop-the-kernel-switching-to-a-new-framebuffer-mode-clearing-output – Ralph Rönnquist Feb 21 '16 at 23:52
  • Regarding local7, it's one of 8 "unreserved" methods (called "facilities") for logging data with syslog. There is no standard that requires or defines which one (local0-7) to use for what purpose. For more info, see [this thread](http://ubuntuforums.org/showthread.php?t=1889743), [this unix.se post](http://unix.stackexchange.com/questions/90842/what-is-the-local6-and-all-other-local-facilities-in-syslog), and [this serverfault post](http://serverfault.com/questions/115923/which-program-defaults-uses-syslog-local0-7-facilities). – Ryan Apr 08 '16 at 18:15
  • Although it won't show it exactly as during the boot up process, have you looked at `dmesg`, which should give you all the technical details you really need. – Julie Pelletier Apr 28 '16 at 05:27
  • 1
    FWIW, i managed to pause the scrolling messages in the right moment with [`Ctrl-S`](https://wiki.archlinux.org/index.php/General_troubleshooting#Console_messages) on NixOS with Linux 4.19. However, it required a lot of luck: i only succeeded in one of about 15-20 attempts. – Alexey Oct 05 '19 at 13:52

1 Answers1

1

You could set a kernel command line argument (something like console=tty0 console=ttyS0,115200n8) to send these to a serial console instead, and then the device that listens on the serial port can simply log it, since then it's just a stream of text.

And systemd is pretty silly if it doesn't log this stuff anyway. Openrc does it in /var/log/rc.log. Also if it wasn't systemd, you could probably modify inittab to just not put a getty/Xorg there on tty1, and prevent anything (like Xorg) from switching elsewhere, and the old text might stay put (just like it did on old pre-systemd openSUSE). Or copy it to another tty (which I think is syslog doing that rather than inittab... and you might see many linux installers doing this on tty9+) If it switches away and back, it just won't scroll back (shift+pgup), but will probably have one page of output. Perhaps someone that knows more about systemd knows the new equivalent to inittab and you can change that.

Peter
  • 1,227
  • 10
  • 9