Ubuntu: How do the md devices get assembled at bootup?

Question

How do the md devices get assembled at bootup in Ubuntu? Is /etc/mdadm/mdadm.conf truly the relevant factor here?

My mdadm.conf is sound and I checked that while I was in the rescue CD environment. When running mdadm -A --scan it finds and assigns the device names as desired. The mdadm.conf contains AUTO -all to take out all automatism from assembling the arrays.

What I need to do is to be able to auto-assemble the md devices as outlined in mdadm.conf at boot time or that when assembling it honors the super-minor value for the 0.9 array and the name (apparently <hostname>:<super-minor>) for the 1.2 arrays and does the right thing without mdadm.conf. What puzzle piece am I missing?

---

I have the following problem. There are two md devices with RAID1 (md0 and md1) and one with RAID6 (md2). I am referring to them by the desired device names. md0 has meta-data version 0.9, the other two have version 1.2. md0 maps to / and the other two are not relevant for booting.

The boot drive is GPT partitioned. There is a glue "BIOS Boot Partition" (sda1) on it. grub-install --no-floppy /dev/sda reports success.

• md0 == sda3 + sdb3
• md1 == sda2 + sdb2
• md2 == sdc + sdd + sde + sdf + sdg + sdh
• sda1 and sdb1 are "BIOS Boot Partition" each

GRUB2 is happy with the /boot/grub/devicemap I gave it and I added part_gpt, raid, mdraid09 and ext2 to the modules to preload in GRUB2.

Since I still had my root volume in the rescue environment, I simply mounted everything and then chrooted into it:

mkdir /target
mount /dev/md0 /target
mount -o bind /dev /target/dev
mount -o bind /dev/pts /target/dev/pts
mount -o bind /sys /target/sys
mount -o bind /proc /target/proc
chroot /target /bin/bash

From there I reset the super-minor on md0 (with meta-data 0.9) and the name on md1 and md2. I also verified that it worked using mdadm --detail .... Other than that I adjusted /etc/default/grub, run update-grub and also grub-install --no-floppy /dev/sda and grub-install --no-floppy /dev/sdb.

After that, when booting, I am always dropped into the initramfs rescue shell, however, because the root file system could not be mounted. The reason, after checking /proc/mdstat appears to be that the respective md device doesn't even get assembled and run. Not to mention that the other two (meta-data version 1.2) drives receive a device number somewhere in the 125..127 range.

Note: GRUB2 comes up from the boot disk. So at the very least it has been embedded correctly. The issue is the transition from the initial rootfs to the proper root file system.

Answer 1

Basic Boot Process

Grub

1. Grub reads its disk, md, filesystem, etc. code from the MBR.
2. Grub finds its /boot partition, and reads the rest of itself out of it. Including the config, and any modules the config specifies need loading.
3. Grub follows the instructions in the config, which typically tell it to load a kernel and initramfs into memory, and execute the kernel.

There is a fallback mode, for when Grub can't actually read the filesystem—either because there wasn't enough space to embed all that code in the boot record, or because it doesn't know the filesystem or layers under it. In this case, GRUB embeds a list of sectors, and reads code from them. This is much less robust, and best avoided. It may even be able to do kernel and initramfs like that (not sure).

Kernel

The kernel then takes control, and does a lot of basic hardware init. This stage is fairly quick. Next, the kernel unpacks the initramfs to a tmpfs, and looks for a /init on that tmpfs. It then executes (in the normal sense, the kernel is fulling running at this point) /init. This is, by the way, a plain old shell script.

Initramfs

You can extract the initramfs by hand by doing something like mkdir /tmp/foo; cd /tmp/foo; zcat /boot/initrd.img-3.8-trunk-amd64 | cpio -idmv.

The initramfs is responsible for loading all the drivers, starting udev, and finding the root filesystem. This is the step that is failing for you—it can't find the root filesystem, so it bails out.

Once the initramfs has finished, it has the root filesystem mounted, and hands over control to /sbin/init.

System boot

At this point, your init takes over—I think Ubuntu is using upstart currently.

What's Broken

I'm not entirely sure what's broken (part, I confess, because I'm much more familiar with how it works on Debian than Ubuntu, though its similar), but I have a couple of suggestions:

• The initramfs has its own copy of mdadm.conf. You may just need to run update-initramfs -u to fix it.
• Look at the boot messages. There may be an error. Get rid of 'quiet' and 'splash' and maybe add 'verbose' to your kernel line to actually see them.
• Depending on storage used, you may need to set the rootdelay parameter.
• When you're dumped to the shell prompt, you don't have a lot of commands, but you do have mdadm. Try to figure out what went wrong. If you fix the problem, boot can continue.

Answer 2

Okay, I found out that I was missing simply one piece. The initrd images had not been updated after tinkering with mdadm.conf.

So what did I do?

I booted into the Ubuntu Server install CD's rescue system. Chose to execute a shell from the installer environment and not to use a root file system. Then (comments prepended with #):

cat /proc/mdstat
# It showed me md125, md126 and md127
# Stop those:
mdadm -S /dev/md125
mdadm -S /dev/md126
mdadm -S /dev/md127
# Assemble the root volume (meta-data version 0.9)
mdadm -Av --update=super-minor --run /dev/md0 /dev/sda3 /dev/sdb3
# Assemble the other two arrays, updating the names (meta-data version 1.2)
mdadm -Av --update=name --run /dev/md1 /dev/sda2 /dev/sdb2
mdadm -Av --update=name --run /dev/md2 /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/sdh
# Check the outcome:
cat /proc/mdstat
# See preferred minor and names:
mdadm --detail /dev/md0
mdadm --detail /dev/md1
mdadm --detail /dev/md2
# All is fine, so proceed ...
# Create directory for the chroot:
mkdir /target
# Mount root volume on it
mount /dev/md0 /target
mount -o bind /dev /target/dev
mount -o bind /proc /target/proc
mount -o bind /sys /target/sys
mount -o bind /dev/pts /target/dev/pts
# Now chroot into it:
chroot /target /bin/bash
# Fix up the GRUB device map to match what 'mdadm --detail' gives as UUID:
nano /boot/grub/devicemap

nano

I'm using nano because vim caused me headaches because of the dumb terminal, literally. You can use Ctrl+x to quit (will prompt to save, Ctrl+k to cut the current line, Ctrl+u to paste a cut line, Ctrl+o to save the buffer.

This sounds complicated, but can be done also with a bash one-liner (albeit a long one):

for i in /dev/disk/by-id/md-uuid-*; do DEV=$(readlink $i); echo "(${DEV##*/}) $i"; done|sort|tee /boot/grub/devicemap

This uses the current names of the md devices and their UUIDs and creates a devicemap for GRUB2's perusal. So assuming the above was done correctly, you should have the correct device names already.

Further on:

# Edit the grub config
nano /etc/default/grub

Make sure it contains:

GRUB_PRELOAD_MODULES="part_gpt raid mdraid09 ext2"

if you have configured your / or /boot partition to be meta-data version 1.2, use mdraid1x instead of mdraid09.

Further:

# Update the initrd images
update-initramfs -c -k all

This above step was the missing link. This apparently makes sure that the mdadm.conf takes effect on boot.

# Install GRUB2 on the two drives eligible for booting, just to be sure
grub-install --no-floppy /dev/sda
grub-install --no-floppy /dev/sdb
# Make the latest config take effect
update-grub

After that leave the chroot and reboot.