Azure VM cloud-init overrides /etc/fstab

Question

I added an additional data disk, as described in the Attach a data disk to a Linux VM documentation.

The partition in question is /dev/sdc1. I added this line to the very bottom. Unfortunately the cloud-init magic works its wonders in a bad way and moves around the entries in /etc/fstab and I end up with a "hidden" mount point.

root@qwerz:/mnt/builds/docker# cat /etc/fstab
# CLOUD_IMG: This file was created/modified by the Cloud Image build process
UUID=cfadafca-7199-49b1-a353-072629b7fcdf       /        ext4   defaults,discard        0 0
LABEL=UEFI      /boot/efi       vfat    defaults,discard        0 0
UUID=0195f789-b5fa-4bea-a91d-dc120bafb23a       /mnt/builds             ext4    defaults,nofail   1   2
#/dev/sdc1      /mnt/builds     ext4    default,nofail  0       0
/dev/disk/cloud/azure_resource-part1    /mnt    auto    defaults,nofail,x-systemd.requires=cloud-init.service,comment=cloudconfig       0       2
root@qwerz:/mnt/builds/docker#

question 1: should I just move my custom point outside of /mnt?
question 2: what's the cloud-init stuff doing anyway and can I get rid of it?

root@qwerz:/mnt/builds/docker# dpkg -l |grep cloud-init
ii  cloud-init                          19.1-1-gbaa47854-0ubuntu1~18.04.1           all          Init scripts for cloud instances
ii  cloud-initramfs-copymods            0.40ubuntu1.1                               all          copy initramfs modules into root filesystem for later use
ii  cloud-initramfs-dyn-netconf         0.40ubuntu1.1                               all          write a network interface file in /run for BOOTIF

From what I understand cloud-init should only run on the very first time the VM boots. However I have the feeling it also runs after every deallocation of any given VM (to save some credits).

The interesting part of cloud-init.log looks like that

2019-06-26 06:01:49,392 - util.py[DEBUG]: Reading from /etc/fstab (quiet=False)
2019-06-26 06:01:49,392 - util.py[DEBUG]: Read 451 bytes from /etc/fstab
2019-06-26 06:01:49,392 - cc_mounts.py[DEBUG]: Attempting to determine the real name of ephemeral0
2019-06-26 06:01:49,392 - cc_mounts.py[DEBUG]: Mapped metadata name ephemeral0 to /dev/disk/cloud/azure_resource
2019-06-26 06:01:49,392 - cc_mounts.py[DEBUG]: changed default device ephemeral0 => /dev/disk/cloud/azure_resource-part1
2019-06-26 06:01:49,392 - cc_mounts.py[DEBUG]: Attempting to determine the real name of swap
2019-06-26 06:01:49,392 - cc_mounts.py[DEBUG]: changed default device swap => None
2019-06-26 06:01:49,392 - cc_mounts.py[DEBUG]: Ignoring nonexistent default named mount swap
2019-06-26 06:01:49,392 - cc_mounts.py[DEBUG]: no need to setup swap
2019-06-26 06:01:49,392 - util.py[DEBUG]: Reading from /proc/mounts (quiet=False)
2019-06-26 06:01:49,393 - util.py[DEBUG]: Read 2608 bytes from /proc/mounts
2019-06-26 06:01:49,393 - util.py[DEBUG]: Fetched {'sysfs': {'fstype': 'sysfs', 'mountpoint': '/sys', 'opts': 'rw,nosuid,nodev,noexec,relatime'}, 'proc': {'fstype': 'proc', 'mountpoint': '/proc', 'opts': 'rw,nosuid,nodev,noexec,relatime'}, 'udev': {'fstype': 'devtmpfs', 'mountpoint': '/dev', 'opts': 'rw,nosuid,relatime,size=8187860k,nr_inodes=2046965,mode=755'}, 'devpts': {'fstype': 'devpts', 'mountpoint': '/dev/pts', 'opts': 'rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000'}, 'tmpfs': {'fstype': 'tmpfs', 'mountpoint': '/sys/fs/cgroup', 'opts': 'ro,nosuid,nodev,noexec,mode=755'}, '/dev/sda1': {'fstype': 'ext4', 'mountpoint': '/', 'opts': 'rw,relatime,discard'}, 'securityfs': {'fstype': 'securityfs', 'mountpoint': '/sys/kernel/security', 'opts': 'rw,nosuid,nodev,noexec,relatime'}, 'cgroup': {'fstype': 'cgroup', 'mountpoint': '/sys/fs/cgroup/memory', 'opts': 'rw,nosuid,nodev,noexec,relatime,memory'}, 'pstore': {'fstype': 'pstore', 'mountpoint': '/sys/fs/pstore', 'opts': 'rw,nosuid,nodev,noexec,relatime'}, 'systemd-1': {'fstype': 'autofs', 'mountpoint': '/proc/sys/fs/binfmt_misc', 'opts': 'rw,relatime,fd=32,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18816'}, 'mqueue': {'fstype': 'mqueue', 'mountpoint': '/dev/mqueue', 'opts': 'rw,relatime'}, 'debugfs': {'fstype': 'debugfs', 'mountpoint': '/sys/kernel/debug', 'opts': 'rw,relatime'}, 'hugetlbfs': {'fstype': 'hugetlbfs', 'mountpoint': '/dev/hugepages', 'opts': 'rw,relatime,pagesize=2M'}, 'fusectl': {'fstype': 'fusectl', 'mountpoint': '/sys/fs/fuse/connections', 'opts': 'rw,relatime'}, 'configfs': {'fstype': 'configfs', 'mountpoint': '/sys/kernel/config', 'opts': 'rw,relatime'}, '/dev/loop0': {'fstype': 'squashfs', 'mountpoint': '/snap/dotnet-sdk/39', 'opts': 'ro,nodev,relatime'}, '/dev/loop1': {'fstype': 'squashfs', 'mountpoint': '/snap/core/7169', 'opts': 'ro,nodev,relatime'}, '/dev/loop2': {'fstype': 'squashfs', 'mountpoint': '/snap/dotnet-sdk/38', 'opts': 'ro,nodev,relatime'}, '/dev/loop3': {'fstype': 'squashfs', 'mountpoint': '/snap/core/6964', 'opts': 'ro,nodev,relatime'}, '/dev/loop4': {'fstype': 'squashfs', 'mountpoint': '/snap/dotnet-sdk/35', 'opts': 'ro,nodev,relatime'}, '/dev/sda15': {'fstype': 'vfat', 'mountpoint': '/boot/efi', 'opts': 'rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro,discard'}} mounts from proc
2019-06-26 06:01:49,393 - util.py[DEBUG]: Writing to /etc/fstab - wb: [644] 451 bytes
2019-06-26 06:01:49,393 - cc_mounts.py[DEBUG]: No changes to /etc/fstab made.
2019-06-26 06:01:49,393 - util.py[DEBUG]: Running command ['mount', '-a'] with allowed return codes [0] (shell=False, capture=True)
2019-06-26 06:01:49,427 - cc_mounts.py[WARNING]: Activate mounts: FAIL:mount -a
2019-06-26 06:01:49,427 - util.py[WARNING]: Activate mounts: FAIL:mount -a
2019-06-26 06:01:49,427 - util.py[DEBUG]: Activate mounts: FAIL:mount -a
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/cloudinit/config/cc_mounts.py", line 495, in handle
    util.subp(cmd)
  File "/usr/lib/python3/dist-packages/cloudinit/util.py", line 2069, in subp
    cmd=args)
cloudinit.util.ProcessExecutionError: Unexpected error while running command.
Command: ['mount', '-a']
Exit code: 64
Reason: -
Stdout: 
Stderr: mount: /mnt/builds: mount point does not exist.

the mount point of course exists, but not in the previously mounted ephemeral0 temporary storage disk mounted at /mnt :(

Why would you mount a temporary disk under /mnt anyway!?

Answer 1

I realize that this is late, but I hope it helps.

In Azure, the linux agent configuration file (waagent.conf) usually sets the mountpoint and you do i from there for most Linux distros supported. You'll usually find the default mountpoint as "/mnt/resource"

In Ubuntu 16.04+ however, cloud-init overrides waagent.conf ephemeral0 mountpoint configuration as well as fstab on every deallocate/start. Still, you can specify the mountpoint in a number of ways that will survive a reboot or deallocate/start. I'll discuss them below.

Now to answer your questions:

question 1: should I just move my custom point outside of /mnt? I think you probably should if you're mounting stuff on /mnt as you probably don't want to mount stuff on a temp. disk unless you intend to do so.

question 2: what's the cloud-init stuff doing anyway and can I get rid of it? I think you should not get rid of it. Instead, you can try to reconfigure the mountpoint the correct way.

How to make it persist? Let's say you want to mount your Azure temp. disk on /mnt/resource.

Method 1 (clean): Create a new cloud-init conf. file in /etc/cloud/cloud.cfg.d to define mountpoints:

    cat >> /etc/cloud/cloud.cfg.d/91-azure_datasource.cfg <<EOF
mounts:
- [ ephemeral0, /mnt/resource ]
EOF

Method 2 (dirty): Remove cloud-init mount options from the temp. disk fstab entry to make it look like this:

/dev/disk/cloud/azure_resource-part1    /mnt    auto    defaults,nofail       0       2

Method 3 (nasty): Look for the mountpoint line in cc_mounts.py, path: /usr/lib/python3/dist-packages/cloudinit/config/cc_mounts.py and change the mountpoint in the code. The line should look like:

defmnts = [["ephemeral0", "/mnt", "auto", defvals[3], "0", "2"],
           ["swap", "none", "swap", "sw", "0", "0"]]

Give it a shot and see how it goes.

Answer 2

the easy workaround that I found, it's just after edit the /etc/fstab file, set chattr +i /etc/fstab, in this way cloud-init is not able to overwrite it