4

I have a VM which has an attached raided device with fstab entry:

/dev/md127 /mnt/blah ext4 nofail 0 2

The raided disks are corrupted and during startup the unit entered emergency/maintence mode, which means only the local host user could exit this mode and start it up normally. During normal startup the following occurred in syslog:

systemd-fsck[1272]: /dev/md127 contains a file system with errors, check forced.
systemd-fsck[1272]: /dev/md127: Inodes that were part of a corrupted orphan linked list found.
systemd-fsck[1272]: /dev/md127: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY.
systemd-fsck[1272]: #011(i.e., without -a or -p options)
systemd-fsck[1272]: fsck failed with exit status 4.
systemd-fsck[1272]: Running request emergency.target/start/replace
systemd[1]: [email protected]: Main process exited, code=exited, status=1/FAILURE
systemd[1]: [email protected]: Failed with result 'exit-code'.
systemd[1]: Failed to start File System Check on /dev/md127.
systemd[1]: Dependency failed for /mnt/blah.
systemd[1]: Dependency failed for Provisioner client daemon.

My guess is that the OS goes to emergency/maintenance mode because of the corrupt raided disks:

systemctl --state=failed
  UNIT                           LOAD   ACTIVE SUB    DESCRIPTION                                            
● [email protected] loaded failed failed File System Check on /dev/md127

What i want is for the VM to startup regardless of whether the raided drives are corrupt/unmountable, so it shouldn't go to emergency/maintenance mode. I followed these posts to attempt at disabling emergency/maintenance mode:

I had to first create the directory local-fs.target.d in /etc/systemd/system/, which felt wrong. I then created a nofail.conf in /etc/systemd/system/local-fs.target.d/nofail.conf containing:

[Unit]
OnFailure=

After loading that drop file, I was able to confirm that the drop file was found by local-fs.target:

sudo systemctl status local-fs.target 
● local-fs.target - Local File Systems
   Loaded: loaded (/lib/systemd/system/local-fs.target; static; vendor preset: enabled)
  Drop-In: /etc/systemd/system/local-fs.target.d
           └─nofail.conf
   Active: active since Tue 2019-01-08 12:36:41 UTC; 3h 55min ago
     Docs: man:systemd.special(7)

BUT, after rebooting, the VM still ended up in emergency/maintenance mode. Have i missed something? Does the nofail.conf solution not work with raided disks?

EDIT: I was able to get a print out of the logs when the system booted to emergency mode (sorry it's a screenshot since i don't have access to the host and had to ask the owner for it):

enter image description here

Here's the output from systemctl for systemd-fsck@dev-md127:

 sudo systemctl status --no-pager --full systemd-fsck@dev-md127
 ● [email protected] - File System Check on /dev/md127
    Loaded: loaded (/lib/systemd/system/[email protected]; static; vendor preset: enabled)
    Active: failed (Result: exit-code) since Thu 2019-01-10 12:05:44 UTC; 2h 57min ago
      Docs: man:[email protected](8)
   Process: 1025 ExecStart=/lib/systemd/systemd-fsck /dev/md127 (code=exited, status=1/FAILURE)
  Main PID: 1025 (code=exited, status=1/FAILURE)

 systemd[1]: Starting File System Check on /dev/md127...
 systemd-fsck[1025]: /dev/md127 contains a file system with errors, check forced.
 systemd-fsck[1025]: /dev/md127: Inodes that were part of a corrupted orphan linked list found.
 systemd-fsck[1025]: /dev/md127: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY.
 systemd-fsck[1025]:         (i.e., without -a or -p options)
 systemd-fsck[1025]: fsck failed with exit status 4.
 systemd-fsck[1025]: Running request emergency.target/start/replace
 systemd[1]: [email protected]: Main process exited, code=exited, status=1/FAILURE
 systemd[1]: [email protected]: Failed with result 'exit-code'.
 systemd[1]: Failed to start File System Check on /dev/md127.

As i pointed out earlier, i have nofail set in /etc/fstab. Now the questions are:

  1. What is the dependency in the failed dependency in the screenshot?
  2. If fsck fails on /dev/md127, why does it enter emergency mode and how do i disable that?

EDIT 2:

A couple of other things i can add are:

  1. the vm is a kvm vm
  2. it's a software raid

Kind regards, Ankur

Ankur22
  • 41
  • 3

2 Answers2

0

Your system tried to repair the corruption and boot automatically, but it could not. That is why it went to emergency mode.

This is not something that you can just override, because the filesystem cannot be mounted in this state, and you specified to mount the filesystem on boot.

You can try one of the following:

  • Don't corrupt the filesystem. Unmount it normally instead of hard powering off the VM while data are still being written to it.
  • Don't mount the filesystem automatically (set noauto in /etc/fstab). The VM will start, but you will still have to manually come in later and mount the filesystem.
  • Switch to a more resilient filesystem, such as XFS.
Michael Hampton
  • 8,658
  • 2
  • 31
  • 54
  • Hi Michael, thanks for the reply. This isn't a critical drive, and i've added nofail option in fstab, so it should boot normally. If it doesn't even boot so that ssh daemon starts, then it becomes extremely difficult to fix this problem in a timely manor. I believe that this option should work. I've tried the same setup with virtualbox and the VM boots up normally even if there are disk errors or the disk is not present, so now it seems that it could be a KVM issue. – Ankur22 Jan 17 '19 at 17:14
  • @Ankur22 That is not what `nofail` does. It means only that the device will be ignored if it does not exist. If the device exists, this does nothing. – Michael Hampton Jan 17 '19 at 17:16
  • 1
    That's interesting. The wording on the man page for systemd.mount suggest something else `nofail With nofail, this mount will be only wanted, not required, by local-fs.target or remote-fs.target. This means that the boot will continue even if this mount point is not mounted successfully.` whereas fstab does indeed suggest it will only ignore incase of missing device `nofail do not report errors for this device if it does not exist.` I know that there used to be a `nobootwait` option in 12.04, is there no similar option to avoid emergency mode? – Ankur22 Jan 17 '19 at 17:53
  • I agree that the behavior is inconsistent with the manpage. `noauto` also doesn't help. :-( – Scott Lamb Mar 21 '20 at 23:22
0

systemd didn't honor nofail on fsck failure due to a bug. I've fixed it in systemd 250 and beyond.

If you're stuck with an older version of systemd, I think your only option is to not use the /lib/systemd/systemd-fsck binary. Maybe don't list the filesystem in /etc/fstab, instead having your own systemd unit to fsck and mount it.

Scott Lamb
  • 1,069
  • 6
  • 7