5

This is a followup to another question.

I figured out something is unmounting my device right after I mount it.

This device is being used by a database (Vertica), which is down and not using the directory while I'm running the mount command.

I'm trying to figure out:

  1. Is systemd the one which unmounts the device?
  2. How can I debug why is that happening?
  3. How do I fix it?

Here's an example of what is happening:

[root@mymachine systemd]# mount -t ext4 /dev/xvdx /vols/data5; ls -la /vols/data5; sleep 5; ls -la /vols/data5
total 36
drwxr-xr-x   5 dbadmin verticadba  4096 Jul 23  2017 .
drwxr-xr-x   9 root    root          96 Jul 16 18:52 ..
drwxrwx--- 503 dbadmin verticadba 12288 Jul 23 13:51 somedb
drwx------   2 root    root       16384 Nov 30  2016 lost+found
drwxrwxrwx   2 dbadmin verticadba  4096 Jun 20 08:32 tmp
total 0
drwxr-xr-x 2 root root  6 Jun  8  2017 .
drwxr-xr-x 9 root root 96 Jul 16 18:52 ..
[root@mymachine ~]#

fstab:

#
# /etc/fstab
# Created by anaconda on Mon May  1 18:59:01 2017
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
UUID=29342a0b-e20f-4676-9ecf-dfdf02ef6683 /                       xfs     defaults        0 0
/dev/xvdb swap swap defaults,nofail 0 0
/dev/xvdy /vols/data ext4 defaults 0 0
/dev/xvdx /vols/data5 ext4 defaults 0 0

Some more logs as per Filipe Brandenburger's suggestion:

Aug 01 16:55:19 mymachine kernel: EXT4-fs (xvdx): mounted filesystem with ordered data mode. Opts: (null)
Aug 01 16:55:19 mymachine systemd[1]: Unit vols-data5.mount is bound to inactive unit dev-xvdl.device. Stopping, too.
Aug 01 16:55:19 mymachine systemd[1]: Unmounting /vols/data5...
Aug 01 16:55:19 mymachine umount[353194]: umount: /vols/data5: target is busy.
Aug 01 16:55:19 mymachine umount[353194]: (In some cases useful info about processes that use
Aug 01 16:55:19 mymachine umount[353194]: the device is found by lsof(8) or fuser(1))
Aug 01 16:55:19 mymachine systemd[1]: vols-data5.mount mount process exited, code=exited status=32
Aug 01 16:55:19 mymachine systemd[1]: Failed unmounting /vols/data5.
Jeff Schaller
  • 66,199
  • 35
  • 114
  • 250
yurmix
  • 291
  • 3
  • 10
  • Can you check output of `journalctl -e` and see if there is something relevant there? Also take a look at `systemctl status /vols/data5`. Usually systemd won't unmount filesystems during normal runtime... Actually, if you mount a new filesystems using `mount` command (outside of systemd), systemd will detect it and create an implicit `.mount` unit for it. So the usual behavior is to "learn" your mounts... But I can't rule out that it's systemd just yet... Start with the logs? – filbranden Aug 01 '18 at 23:52
  • I must say `systemd` not having an option to not unmount unused mountpoints after a while is really becoming an hassle. – Scrooge McDuck Dec 03 '22 at 10:23

1 Answers1

5

Ok, that was in interesting debugging experience... Thanks Filipe Brandenburger for leading me to it!

  1. Is systemd the one which unmounts the device?

Yes. journalctl -e shows a related message:

Aug 01 16:55:19 mymachine systemd[1]: Unit vols-data5.mount is bound to inactive unit dev-xvdl.device. Stopping, too.

Apparently I'm not the first one to encounter it. See this systemd issue: systemd umounts manual mounts when it has a failed unit for that mount point #1741

  1. How can I debug why is that happening?

Run journalctl -e for debugging.

  1. How do I fix it?

This workaround worked for me: run the command below, then try mounting again.

systemctl daemon-reload

That's all, folks!

yurmix
  • 291
  • 3
  • 10
  • Which distro and version of systemd are you using? Also, can you expand on why you copied the device? (You mentioned copying `xvdl` to `xvdx`...) I'd actually recommend you should mount your filesystems using UUID= or LABEL=, that's probably more robust... Anyways, glad you managed to solve it! – filbranden Aug 02 '18 at 02:21
  • CentOS 7.3.1611 (Core) with systemd 219. I switched AWS EBS volumes and used rsync to copy (since it wasn't a lot of data). I believe I have used UUIDs in the past but not on this server. BTW, It had a different issue of mount mixup on boot, but that's a different story :) – yurmix Aug 02 '18 at 05:18
  • Yeah that's probably older than the fix went into systemd... Though Red Hat backports many fixes, so really it's hard to tell... – filbranden Aug 02 '18 at 05:19
  • I had this very issue on an up to date Debian 9, but your answer fixes it. Interesting situation. Also @frostschutz pointed me to this still open issue: https://github.com/systemd/systemd/issues/1741 which seems to discuss if and how to fix it. But apparently this issue is present since 2015. – Thorian93 Oct 18 '19 at 05:45