5
Distro: Gentoo
systemd: 242

For a while, on both of the systems I have here, I've been unable to do a proper shutdown (halt, poweroff, reboot). It gets most of the way through shutting down services and such, and then just stops. I have to use magic SysRq keys to finish the job.

I finally enabled debug-shell.service and took a look. It turns out, systemd is hanging at 100% CPU at some point. There are usually 1 or 2 services that still have processes, some of which are zombies (I assume because systemd would normally remove them but isn't working).

This is about all I've been able to determine. What else can I do to diagnose the issue?

EDIT: Example screenshot (cropped): https://i.stack.imgur.com/pZO2o.jpg The contents actually vary considerably, as the order of things may be different.

EDIT 2: Relevant part of the syslog. I don't see much interesting in there. The only thing that was "stopping" without a "stopped" was Dovecot. However, you can see at the bottom, it did finally exit. This seems to have been just before systemd started hanging. I remember from the debugging shell, Dovecot was left as a zombie. Just a little while ago, I tried shutting down Dovecot (successfully) and then doing the reboot, but systemd still hangs.

ddawson
  • 313
  • 2
  • 9
  • can you give a sample from the output during shutdown , problematic procceses must be displayed there. – Angel May 05 '19 at 13:17
  • Not sure how to copy text under those conditions, so I've settled for a screenshot. – ddawson May 05 '19 at 14:39
  • After a shutdown/reboot, do `systemctl -b -1` to see what got logged during shutdown of the previous session before you forced it. – Mio Rin May 05 '19 at 14:44
  • Do you find something like "Stopping xy" without matching "Stopped xy" in your output? That would be a hint. Also: Sometimes there is a long timeout, like 90 seconds or more. Please wait 10 minutes then watch which may have taken so long. – Ned64 May 05 '19 at 14:46
  • I forgot to mention: the way I first encountered this was that I started a shutdown, and then left for a few hours. It was still sitting there in the middle of the shutdown when I returned. So I think I've already ruled out a timeout from the start. – ddawson May 05 '19 at 14:54
  • @Mioriin ITYM `journalctl`. I'll look into it when I get more time. But the fact that the systemd process is running at 100% CPU is strange. – ddawson May 05 '19 at 16:43
  • @ddawson Ah, yes. I meant `journalctl`. Sorry. – Mio Rin May 05 '19 at 17:58
  • Added syslog from where shutdown started. – ddawson May 06 '19 at 21:08

2 Answers2

3

It indeed looked like this bug which affects the initial release of systemd 242: https://bugs.gentoo.org/685002 (upstream: https://github.com/systemd/systemd/issues/12335).

And, in fact, the change from 242 to 242r1 (in gentoo's versioning) was to apply this patch: https://gitweb.gentoo.org/repo/gentoo.git/commit/sys-apps/systemd?id=eb1d80e6a30d09f9f139877c5b754c8a8e918d7a

lbonn
  • 248
  • 2
  • 6
1

Well, after I updated to 242-r1, this is no longer happening. I was able to do a proper shutdown. Unfortunately, I am not sure why. It may be something fixed in systemd, or a dependency somewhere.

Rakib Fiha
  • 580
  • 3
  • 9
  • 24
ddawson
  • 313
  • 2
  • 9