Where is /cow for Linux booted from LiveUSB?

Question

I wanted to see what files are added on top of ISO 9660 when LiveUSB Linux is running. When booted with persistence upper and work folders are on USB drive clearly seen. I run mount on Linux booted from LiveUSB "usual way" (w/out persistence) and saw / is mounted via overlayfs and upperdir=/cow/upper. But sudo ls /cow gives no such file or directory.

Where is /cow and how to see its contents?

Added 1: I was able to extract contents of initrd from liveUSB via unmkinitramfs (see https://unix.stackexchange.com/a/495524/446998)

$ find . -type f -exec bash -c 'cat {} | grep "/cow/upper" && ls -l {}' \;
    if [ ! -d /cow/upper ]; then
        mkdir -p /cow/upper
            /cow/lost+found|/cow/upper|/cow/log|/cow/crash|/cow/install-logs-*)     continue ;;
            mv "$cow_content" /cow/upper
    mount -t overlay -o "upperdir=/cow/upper,lowerdir=$mounts,workdir=/cow/work" "/cow" "$rootmnt" || panic "overlay mount failed"
-rw-r--r-- 1 alex alex 33834 Jun 24  2020 ./main/scripts/casper  

Next step I envision is to understand how /cow is created because is not see in contents of initrd

Answer 1

LiveUSB sticks use an overlay file system such as overlayfs. There are a couple of different ones so I'm not certain which it uses but the principle is pretty much the same in each.

The root file system that you see is composed of layers. For a LiveUSB this will be

• an upper writable layer
• a read-only lower layer containing your operating system image and often stored in a squashfs file.

Initially you see just the files in the lower layer because the upper (writable) layer is empty. When you write new files they will be written to the upper layer. But other changes such as moving a file and editing a file will actually cause the file to be copied to the upper layer from the lower layer.

The upper layer can be a few things. If the LiveUSB doesn't have persistence it's often a tempfs stored only in RAM. However if there's persistence it may be something different... perhaps it's a writable file formatted with EXT4.

If it's a writable file then you may be able to find that file and mount it separate to the overlayfs to poke around what's inside.

---

Where is /cow and how to see its contents?

That's the most tricky part of this question to answer. The fact is it may not exactly exist anymore, at least not under that name.

It did exist during the execution of initramfs. But then initramfs called pivot_root which would have moved it... and then most likely tried to unmount it so now no userland process can actually see it any more. The Kernel must logically still have it mounted though.

If it was mounted with persistence it may be possible to discover exactly what /cow was and re-mount it elsewhere.

Answer 2

// Experience based on kubuntu 22 lts livecd

after chroot

the last step of ramdisk (/cdrom/casper/initrd) is
run-init {rootmnt}" "${init}" "$@"

which do something like
chroot {rootmnt}" "${init}" "$@"

after that step it may affect the observation of the original mount point.

before chroot

Fortunately there are ways to pause at an interactive shell before chrooting (press ctrl+d or exit to continue)

kernel boot cmdline args
break=top,premount,mount,mountroot,bottom,init
may do the trick.

//BTW: manjaro 22 only support break=premount (same as break=y) or break=postmount, not support multiple values compound with ,

also another cmdline args might help
debug or debug=y
which turn on detail log during ramdisk running

// boot args can be edit at grub menu by press e
// those information comes from reading the ramdisk script

in ramdisk

you already unmkinitramfs and found scripts/casper
which handle casper-rw persistence things

./scripts/casper

setup_overlay() {
    image_directory="$1"
    rootmnt="$2"

    # Mount up the writable layer, if it is persistent then it may well
    # tell us what format we should be using.
    mkdir -p /cow
    cowdevice="tmpfs"
    cow_fstype="tmpfs"
    cow_mountopt="rw,noatime,mode=755"

    # Looking for "$(root_persistence_label)" device or file
    if [ -n "${PERSISTENT}" ]; then
        cowprobe=$(find_cow_device "$(root_persistence_label)")
        if [ -b "${cowprobe}" ]; then
            cowdevice=${cowprobe}
            cow_fstype=$(get_fstype "${cowprobe}")
            cow_mountopt="rw,noatime"
        else
            [ "$quiet" != "y" ] && log_warning_msg "Unable to find the persistent medium"
        fi
    fi

    mount -t ${cow_fstype} -o ${cow_mountopt} ${cowdevice} /cow || panic "Can not mount $cowdevice on /cow"

code here determine the persistence partition, and mount it

keep /cow after chroot

in ramdisk shell, after /cow /root ready

mkdir /root/_cow
mount -o bind /cow /root/_cow

then after chroot into system, /_cow can access the original outer /cow

Answer 3

I've discovered this before the answer my @yurenchen, which gives this idea at the end.

Below particular command edits casper script adding bind mount for shownmounts boot option. One need to unpack and repack initramfs (initrd file for Linux Mint 21) for that. After that /cow's layers are visible in /casper/cow after liveUSO system completed boot.

# add mount of /cow
sudo sed --in-place --regexp-extended -- 's|(if [[] -n "[$][{]SHOWMOUNTS[}]" []]; then)|\1\n\n        mkdir -p "${rootmnt}/${LIVE_MEDIA_PATH}/cow"\n        mount --bind /cow "${rootmnt}/${LIVE_MEDIA_PATH}/cow"\n|' $work_path/initrd/main/scripts/casper

# change mount option from "move" to "bind".
sudo sed --in-place --regexp-extended -- 's|(mount )-o move( "[$][{]d[}]" "[$][{]rootmnt[}]/[$][{]LIVE_MEDIA_PATH[}]/[$][{]d[#][#][*]/[}]")|\1--bind\2|' $work_path/initrd/main/scripts/casper

Also, as a bonus second command changes type of mount as for some reason in booted system in /casper I saw folder for squashfs, but it was empty. With with command it was not, but filled with contents. P.S. /rofs has same, but for only one upper layer, there could be several squashfs files in the distro's ISO.