14

My desktop computer running Debian Jessie started dropping into an emergency mode shell on every boot up. The screen says to use journalctl -xb to find the reason, and to use systemctl default to continue booting. When I execute systemctl default, the system continues to boot, and after a couple weeks of using the system there is nothing apparently wrong.

Looking through journalctl -xb, nothing stands out as being the reason for dropping to an emergency shell. Is there an easy way to determine exactly the reason it decided to go into emergency mode? Are there other flags or bootup options that will make it obvious where the issue is?

sourcejedi
  • 48,311
  • 17
  • 143
  • 296
jordanm
  • 41,988
  • 9
  • 116
  • 113
  • 2
    It should be visible in the journal but with the limited information you provide, there is no way to guide you. Do you have a copy of `journalctl -xb ` when it happened? – Julie Pelletier Nov 28 '16 at 06:33
  • 3
    Boot in verbose logging mode `systemd.log_level=debug systemd.log_target=kmsg log_buf_len=1M` for forensic level detail... – jasonwryan Nov 28 '16 at 06:43
  • 2
    Existing logs should already give you the reason. There are so many reasons that it is very difficult to guess. – Giacomo Catenazzi Nov 28 '16 at 11:16
  • 1
    I have the same problem on an instance of Ubuntu 16.04 . Been installing, using, repairing a lot of Linux systems for the past 20 years. Nothing special on screen and this time nothing stands out in the logs. Screen says Ctrl-D to continue booting, but that only leads back to the same prompt after a moment. Without a clue. Frustrating, isn't it? – Stéphane Gourichon Mar 12 '17 at 12:20
  • 1
    Have you tried all the steps from the "Diagnosing Boot Problems" section in [Debugging systemd](https://freedesktop.org/wiki/Software/systemd/Debugging/)? – Timothée Ravier Jul 08 '17 at 11:47

3 Answers3

12

The failure should have shown a red [ FAIL ] on the console (instead of [ OK ]), with the unit's description next to it. Typically the first failures are most important. Use shift+pageup on the console to scroll up and view the past few screenfuls of output. This might not work if there's too much output.

This works even if you don't normally see [ OK ] messages, e.g. due to quiet on the kernel command line as used by Debian. On the first failure, systemd switches to verbose mode.

Otherwise, you can use systemctl. Without any options, it shows a massive list of known units with failures highlighted in red. To show just the failed ones, use systemctl --state=failed or systemctl --failed.

If you search through the unit files, there are only a very few ways for the boot to fall back to emergency.target. It's usually when a .mount unit for a local filesystem fails, causing local-fs.target to fail. Or when your initramfs fails to mount the root filesystem, if your initramfs uses systemd.

local-fs.target has OnFailure=emergency.target. And it gets failed because units for local filesystems are automatically added to the Requires list of local-fs.target (unless they have DefaultDependencies=no).

$ systemctl show --property Requires local-fs.target
Requires=-.mount home.mount boot.mount boot-efi.mount
sourcejedi
  • 48,311
  • 17
  • 143
  • 296
  • Thank you! Although other answers were useful in the way of "yes, I've tried/looked/done that" check-list, this answer was the most precise on "why". Others answered in the generic and abstract ways, for there are (probably) many ways to cause it; but this answer was most precise on WHERE to begin! – HidekiAI Dec 31 '21 at 17:23
2

Every once in a while I run into a "maintenance mode" prompt, and I have to scroll through journald for errors as well. Since journalctl uses less as the pager, you should be able to apply any less shortcuts to your search.

Typically, I would rely on the search function (/) and search for anything equivalent to "error", "warning", or "fail". And make sure to -i to force case insensitive search.

So my keystrokes would tend to look like:

-i (case insensitive)
g (move to start)
/error
nnnn (skip through results)
g (move to start)
/fail
nnnn (skip through results)
g (move to start)
/warn
nnnn (skip through results)

It's technically not an exhaustive or exact search for the precise problem, but I've never missed a boot issue this way.

Some related less keyboard shortcuts below:

http://www.thegeekstuff.com/2010/02/unix-less-command-10-tips-for-effective-navigation/

madumlao
  • 1,686
  • 10
  • 8
  • I believe it should also be possible to page through quickly and look for the red messages (LOG_ERR and above). `systemd` will log red messages for failures to start a service unit, or more importantly failure to mount a filesystem. – sourcejedi Dec 09 '17 at 12:04
0

Sometimes journalctl -xb use of "less" as pager gets in the way of speedy discovery of the problem. I instead use:

journalctl -xb | egrep -i '(error|fail|warn)'

Also /var/log/boot.log is a copy of the non-quiet bootup sequence, easy to see color-coded problems with:

more /var/log/boot.log

I use "more" instead of "less" because the default behavior of the latter is to escape the control sequences, which changes the color-coding to (ahem) useless mud.

TCB13
  • 721
  • 1
  • 9
  • 26
Kevo
  • 1
  • 1