32

It's a common scenario. For whatever reason, the initramfs (OpenSUSE, in case it matters) has failed to find the root filesystem, so it drops you into a rescue shell. I know perfectly well what device needs to be mounted though. My question:

What is the correct procedure to mount the root filesystem and continue the boot sequence?

Presumably that's the whole point of the rescue console. And yet, nobody seems to have documented how you actually do this.

Obviously I can mount the root filesystem somewhere. But how do I make that the root of the filesystem tree? And now do I continue the normal boot process after that? (I thought just exiting the shell would do it... but it doesn't.) What exactly do you need to get mounted before you continue, and how do you continue?

Gilles 'SO- stop being evil'
  • 807,993
  • 194
  • 1,674
  • 2,175
MathematicalOrchid
  • 5,664
  • 7
  • 35
  • 62
  • 2
    I could be wrong, but I think once you're in a rescue shell you can't continue the current boot, you fix things so the next boot will succeed – Eric Renouf Jun 23 '16 at 14:14
  • @EricRenouf manually booting the full system may very well be the easiest way to fix such an issue. I had problems in the past where the initramfs failed to open my cryptsetup-luks encrypted root partition, and the easiest fix was to manually boot it and then run `update-initramfs -u`. I absolutely couldn't get it working when I just `chroot`ed into the root filesystem from a rescue system; the resulting initramfs was always broken. – Martin von Wittich Jun 23 '16 at 15:13
  • Is it actually an initrd, or an initramfs? (Just because the file is called `initrd` doesn't mean that it's one: most distributions have switched to `initramfs` but keep calling the file `initrd`.) **What distribution are you using** (as what the initrd/initramfs does depends on what the distribution put there)? – Gilles 'SO- stop being evil' Jun 23 '16 at 22:40
  • @Gilles I'm pretty sure it's actually initramfs. Not sure if it actually makes a difference though; either way, I've got a mini filesystem and I need to mount the real filesystem. (OpenSUSE, in case it matters.) – MathematicalOrchid Jun 24 '16 at 07:57

5 Answers5

11
exec switch_root /mnt/root /sbin/init

https://wiki.gentoo.org/wiki/Custom_Initramfs#Init

Petr Ketner
  • 191
  • 2
  • 3
8

It depends on the kind of problem, if the problem was borking the initramfs image itself then you'll need to actually regenerate it (run update-initramfs) to fix things. Remember the initramfs file system is a RAM filesystem, so to fix anything you need to fix the compressed initramfs image and/or fix the root filesystem.

I've used the following procedure on an LUKS encrptyed Ubuntu system in order to resume a boot broken by a bad crypttab.

First, I decrypted the partition

cryptsetup luksOpen /dev/nvme0n1p3 nvme0n1p3_crypt

Note the device_crypt isn't random. It needs to match the name that's normally used when your system mounts (partition + _crypt seems to be standard).

Next, you need to activate the Volume Groups on that now-decrpyted partition.

vgchange -ay

Now, you can try mounting the file systems, /, /boot, proc, swap, etc, and getting things running that way.

Though in my case all I had to do was type exit and initramfs saw the logical volumes sitting there and happily resumed the boot. At that point it was easy to fix the damange and run update-initramfs -u

aluchko
  • 81
  • 1
  • 2
  • Where do you mount /, /boot, etc *to*? I think that was the question - and it's certainly mine, stuck as I am in a very similar situation. – Diagon Nov 19 '22 at 14:23
  • @Diagon, you can mount the /, /boot etc. in any directory in the initramfs. Then, you can go inside that directory to find all your files and make necessary fixes. – Sadman Sakib Jan 08 '23 at 20:35
  • 1
    @SadmanSakib / thanks, but once mounted, is there a way to then continue the boot sequence? – Diagon Mar 23 '23 at 05:22
  • @Diagon, the way to continue boot sequence depends on the issue. In my case, my default grub menu had to be corrected. So, I opened `grub.cfg` with vim, made changes and rebooted the computer. – Sadman Sakib Mar 25 '23 at 13:53
  • 1
    @Diagon On a Ubuntu 22.04, I had to mount on `/root`. But when the problem is not too deep, you just need to setup cryptsetup/lvm/etc. and exit from the initramfs prompt (with Ctrl+D or `exit`) so that the init scripts can find the devices of /etc/fstab and pick up from there. As a memento, here are a few of the commands I used : `cryptsetup open /dev/sda5 mycrypt ; lvm lvs ; mount -t auto /dev/ubuntu-vg/root /root ; mount -t auto /dev/sda1 /root/boot ; mount -t proc none /root/proc ; mount -t sysfs none /root/sysfs` – Chl Jul 29 '23 at 10:52
  • @Chl / Useful, thank you. – Diagon Jul 31 '23 at 13:05
5

Simply run fsck command without options from initramfs then reboot

eg: the root partition is sda3

fsck /dev/sda3
GAD3R
  • 63,407
  • 31
  • 131
  • 192
  • The reboot just dropped me to grub shell instead of initramfs shell. Now I'll do [everything](https://www.linux.com/learn/how-rescue-non-booting-grub-2-linux) over again to get to initramfs shell – jaam Feb 24 '19 at 21:52
  • 1
    This worked for me. `reboot` didn't do anything, but simply running `exit` after completing the checks continued the booting process, now with a good file system. – janh Jul 17 '20 at 21:39
  • I had a similar [problem](https://unix.stackexchange.com/questions/648824/stuck-at-busy-box-attempt-to-boot-old-linux-at-another-partition) running fsck give clean. But I still won't able to boot normally. – Adi Prasetyo May 10 '21 at 17:37
2

If the root= parameter is correct and the issue is just that the necessary device isn't available (for example because the initramfs failed to assemble an md RAID), then it's enough to make the device available manually, for example:

mdadm --assemble ...

Then check that the device is there and if everything looks good, hit ctrl + d or type exit to quit the initramfs shell. The initramfs will then mount the root filesystem and continue to boot as usual. After the system has booted, you should then repair the underlying issue, for example by running update-initramfs -u.

I haven't yet found a way to tell the initramfs to boot a different device than what /proc/cmdline says. Maybe someone else has an idea?

Martin von Wittich
  • 13,857
  • 6
  • 51
  • 74
  • i'd gotten stuck due to a recuse cd hard coding a specific md number, md127, when the booting kernel assigned the number md0, i got out of this was a symlink and an update-grub + update-initramfs afterward restoring the UUID behavior. `# ln /dev/md0 /dev/md127; exit` – ThorSummoner Oct 13 '22 at 07:22
1

normal procedure is

  1. mount /dev/sdX /mnt
  2. fix problem on /mnt
  3. reboot

You might want to

  1. mount /dev/sdX /mnt
  2. fix /mnt
  3. umount /mnt
  4. mount /dev/sdX /
  5. finish boot manualy

This is not recommended, you'll have to do it on every boot. In a production environment, you can't be sure manual boot follow same steps as automatic one.

However in a emergency with critical data, step 5 usualy boil down to:

  • 5.1 set up network
  • 5.2 copy important file to safe place
Archemar
  • 31,183
  • 18
  • 69
  • 104