systemd: serialize boot

Question

Booting on my Ubuntu 17.04 fails, and I want to debug it.

The booting fails randomly and I believe it is due to a race condition.

Can I ask systemd not to parallelized any tasks, so I can see if this causes the boot to fail predictably?

Answer 1

A workaround: run services manually

I had a crash during startup. In my case, the crash was after reaching the basic.target target, but before multi-user.target, so I wanted to find out which of the services started by multi-user.target was causing the crash.

First I arranged to boot to basic.target plus a root shell. You can do that permanently (assuming you manage to boot at all!) with

systemctl set-default basic.target
systemctl enable debug-shell

The debug-shell service runs a root shell on tty 9.

You can get the same effect by adding the parameters systemd.unit=basic.target systemd.debug-shell to the kernel command line. For example, in Grub, edit the command line to something like

linux /vmlinuz-4.13.0-38-generic root=/dev/mapper/crypt-root ro systemd.unit=basic.target systemd.debug-shell

From this shell, I ran the following script to start services one by one. Note that this is largely untested (I ran it once, and it crashed as expected on the problematic service).

#!/bin/sh
wants=$(systemctl show -p Wants multi-user.target | sed 's/^Wants=//' | tr ' ' '\n' | sort)
log=/var/tmp/multi-user-steps-$(date +%Y%m%d-%H%M%S)

log () {
  echo "$* ..." | tee -a "$log"
  sync
  "$@"
  ret=$?
  echo "$* -> $ret" | tee -a "$log"
  sync
  return $ret
}

# systemd services
for service in $wants; do
  log systemctl start $service
  sleep 2
done

# upstart services
for conf in /etc/init/*.conf; do
  service=${conf##*/}; service=${service%.conf}
  log service ${service} start
  sleep 2
done

# sysvinit services
for service in /etc/rc3.d/S*; do
  log ${service} start
  sleep 2
done

Add extra dependencies

The script below declares “Before” dependencies on the systemd units that are direct dependencies of a given target in order to force them to run in a specific order. You may want to run it on multi-user.target or basic.target.

Note that this script doesn't work in general because it doesn't take existing dependencies into account: it may cause a dependency loop. A proper script should collect existing dependencies and make a topological sort. I've solved my problem so I don't intend to work on it anymore; I'm posting it in case someone wants to adapt it to their needs.

Note also that this does not affect Upstart and SysVinit services.

Make a backup of /etc before running this! (I strongly recommend using etckeeper.)

#!/bin/sh
set -e

if [ $# -eq 0 ] || [ "$1" = "--help" ]; then
  cat <<EOF
Usage: $0 TARGET
Linearize the dependencies of a systemd target so it starts deterministically.
This scripts adds systemd unit files called linearize-for*.conf containing
extra Before= dependencies for each dependency of TARGET.
EOF
fi

service_dir=/etc/systemd/system
target=$1

wants=$(systemctl show -p Wants "$target" | sed 's/[^= ]*=//' |
                                            tr ' ' '\n' | sort)
previous=
for want in $wants; do
  [ -d "$service_dir/$want.d" ] || mkdir "$service_dir/$want.d"
  cat <<EOF >"$service_dir/$want.d/linearize-for-${target%.*}.conf"
[Unit]
Before=$previous
EOF
  previous=$want
done