2

By mistake, I deleted an lxc image file. The container is still running and the file is therefore not yet actually deleted until I stop the container. I'd like to avoid stopping the container as it is quite sensitive.

I tried to find the deleted file with: for i in $(ls /proc/|grep '^[0-9]*$'); do ls -l /proc/$i/fd|grep delete; done But this doesn't find my loop device. Same with a simple lsof | vm-

If I run lsof on another image that is not delete, it doesn't show me any process using it: lsof /var/lib/vz/images/100/vm-100-disk-0.raw. Probably because it's open by the kernel, not a process.

As suggested in comment:

# losetup -l
NAME       SIZELIMIT OFFSET AUTOCLEAR RO BACK-FILE                                          DIO
/dev/loop1         0      0         1  0 /var/lib/vz/images/200/vm-200-disk-0.raw (deleted)   0
/dev/loop0         0      0         1  0 /var/lib/vz/images/100/vm-100-disk-0.raw             0

I tried:

debugfs /dev/mapper/pve-data
debugfs:  cd images/200
debugfs:  lsdel
 Inode  Owner  Mode    Size      Blocks   Time deleted
 0 deleted inodes found.

I guess that's because it's not deleted yet. It is a bit risky to just let it get deleted and hope that it appears here and doesn't get corrupted (it's >300Gb)

Inside the container, mount gives:

/var/lib/vz/images/200/vm-200-disk-0.raw (deleted) on / type ext4 (rw,relatime,data=ordered)

Any solution apart from dumping the entire filesystem and recreating the container entirely ? (Also, the host drive is almost full, I don't have enough space right now to create a second container next to it. I'm afraid that downsizing the storage would actually result it the actual deletion. :(

Eric Darchis
  • 123
  • 6
  • You have different names in `lsof` and the `mount` output. – Mathias Weidner Dec 18 '19 at 18:48
  • 1
    `lsof` won't tell you about the backing files of the loop devices. `losetup -l` will tell you that. `cat /proc/self/mountinfo` inside the container will tell which loop device is mounted on `/`. But that's only half of the story -- if you have enough space on the disk and want just to recover the data, you can just `cat /dev/loopX /path/to/save`. If you want to resurrect a file on a live system/partition ... I have no idea how you could do that. –  Dec 18 '19 at 19:01

2 Answers2

3

[not a complete answer, but too long to put in a comment]

You can find the inode of a (possibly deleted) backing file of a loop device with the LOOP_GET_STATUS or LOOP_GET_STATUS64 ioctls: it's the .lo_inode field of the loop_info and loop_info64 structs.

As I wasn't able to find any command line utility exposing that info, here is perl one-liner that should do it:

perl -le 'ioctl STDIN, 0x4C05, $s = pack "a512" or die "ioctl: $!"; print unpack "x[Q]Q", $s' </dev/loop1
1179684

More info in the loop(4) manpage and in the /usr/include/linux/loop.h file.

But I don't know if there's any safe way to resurect a deleted file by its inode: I don't think that you can use debugfs(8) on a mounted live file system without corrupting it beyond repair, and there's no way to create a link to a deleted file.

The only safe way I can think of is to copy the whole loop device / partition while it's still live:

cp --sparse=always /dev/loop1 /path/where/to/save/it
  • I think that you're right and the debugfs approach is bound to do more harm than good. I appreciate the perl script . debugfs seems confused about that inode too, so it's better to leave it. The cp of /dev/loop1 is also an interesting idea. I'll give it a try. – Eric Darchis Dec 18 '19 at 21:27
0

As long as the container is running, the file shouldn't be deleted from hard disk.

You can find the inode with lsof and possibly recover the file with debugfs as described in the answer from Andrew Gallagher to this question.

If you can mount an external disk to your server, you can try the accepted answer and copy the file there.

Mathias Weidner
  • 669
  • 4
  • 10