Restore files removed with rm (even from a LUKS-encrypted disk)

Question

The problem

I erroneously removed several files from my /home/username with rm. I realized the mistake as soon as I hit enter, but the damage was done. I immediately created a full disk image with sudo dd if=/dev/sda of=/media/username/external_drive/image.iso and copied it to another PC and prepared to follow a very long path towards data recovery. And then realized I had no idea about where to start from.

What I did

I read some guides online and eventually extundelete /dev/partition_to_recover_from --restore-directory /path/to/restore came up as the most promising solution, so I tried it.

The first problem I encountered was that I had encrypted my drive with LUKS (during OS install) and had to decrypt it. After some more research, I prepared the partition with the following commands (here I changed the real volume group name from the real value of <my_pc_name>-vg to pc-vg).

$ sudo kpartx -a -v image.iso # map the disk image partitions
add map loop0p1 (254:0): 0 997376 linear 7:0 2048
add map loop0p2 (254:1): 0 2 linear 7:0 1001470
add map loop0p5 (254:2): 0 975769600 linear 7:0 1001472

$ sudo cryptsetup luksOpen /dev/mapper/loop0p5 img # unlock the partition with my data
Enter passhprase for /dev/mapper/loop0p5:

$ lsblk
NAME                  MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
loop0                   7:0    0 465,8G  0 loop
├─loop0p1             254:0    0   487M  0 part
├─loop0p2             254:1    0     1K  0 part
└─loop0p5             254:2    0 465,3G  0 part
  └─img               254:3    0 465,3G  0 crypt
    ├─pc--vg-root   254:4    0 464,3G  0 lvm
    └─pc--vg-swap_1 254:5    0   980M  0 lvm

[...omitting other lsblk output...]

$ sudo vgchange -a y pc-vg
  2 logical volume(s) in volume group "pc-vg" now active

and then tried to recover with

$ sudo extundelete /dev/mapper/pc--vg-root --restore-directory /home/username/path/to/restore
NOTICE: Extended attributes are not restored.
WARNING: EXT3_FEATURE_INCOMPAT_RECOVER is set.
The partition should be unmounted to undelete any files without further data loss.
If the partition is not currently mounted, this message indicates 
it was improperly unmounted, and you should run fsck before continuing.
If you decide to continue, extundelete may overwrite some of the deleted
files and make recovering those files impossible.  You should unmount the
file system and check it with fsck before using extundelete.
Would you like to continue? (y/n)

However, the partition was not mounted and df confirmed that. Also, sudo fsck -N only wanted to operate on /dev/sdaX. In doubt, I rebooted the system and repeated the above steps. I received exactly the same output, and considering that I was working on a copy of the original disk image (so I had a backup to use in case of data loss) this time I answered y. The result was:

$ sudo extundelete /dev/mapper/pc--vg-root --restore-directory /home/username/path/to/restore
NOTICE: Extended attributes are not restored.
WARNING: EXT3_FEATURE_INCOMPAT_RECOVER is set.
The partition should be unmounted to undelete any files without further data loss.
If the partition is not currently mounted, this message indicates 
it was improperly unmounted, and you should run fsck before continuing.
If you decide to continue, extundelete may overwrite some of the deleted
files and make recovering those files impossible.  You should unmount the
file system and check it with fsck before using extundelete.
Would you like to continue? (y/n) 
y
Loading filesystem metadata ... extundelete: Extended attribute has an invalid value length when trying to examine filesystem

I did do other research, but I couldn't understand what that means.

The questions

I'll try to avoid the XY problem.

Is the method I used to try to recover my data corect? If so, what is extundelete complaining about and how can I resolve it? If not, how can I (try to) restore my data from the LUKS-encrypted disk in Debian?

If any additional info is required, please ask for it.

P. S.: «Restore from your recent backup you obviously have» is the correct answer, I know =).

I do have a full backup of my home taken a couple of days before the data loss (not such a n00b), but I lost the product of more than twenty hours of work and I would like to have it back.

---

Update

I tried running fsck on the partition with my data, and the result was

$ sudo fsck -r /dev/mapper/pc--vg-root
fsck from util-linux 2.36.1
e2fsck 1.46.2 (28-Feb-2021)
/dev/mapper/pc--vg-root: recovering journal
Clearing orphaned inode 7077927 (uid=1000, gid=1000, mode=0100600, size=0)
Clearing orphaned inode 7077925 (uid=1000, gid=1000, mode=0100600, size=65536)
Clearing orphaned inode 19794062 (uid=1000, gid=1000, mode=040775, size=4096)
Clearing orphaned inode 18366502 (uid=1000, gid=1000, mode=040755, size=4096)
Clearing orphaned inode 18366515 (uid=1000, gid=1000, mode=040755, size=4096)
Clearing orphaned inode 18366503 (uid=1000, gid=1000, mode=040755, size=4096)
Clearing orphaned inode 18366504 (uid=1000, gid=1000, mode=040755, size=4096)
Clearing orphaned inode 18366511 (uid=1000, gid=1000, mode=040755, size=4096)
Clearing orphaned inode 18366512 (uid=1000, gid=1000, mode=040755, size=4096)
Clearing orphaned inode 18351755 (uid=1000, gid=1000, mode=0100444, size=15383322)
Clearing orphaned inode 18351757 (uid=1000, gid=1000, mode=0100444, size=12832)
Clearing orphaned inode 18366521 (uid=1000, gid=1000, mode=040755, size=4096)
Clearing orphaned inode 7078039 (uid=1000, gid=1000, mode=0100600, size=0)
Clearing orphaned inode 7077945 (uid=1000, gid=1000, mode=0100600, size=65536)
Clearing orphaned inode 11927591 (uid=0, gid=0, mode=0100644, size=147932)
Clearing orphaned inode 18096551 (uid=0, gid=0, mode=0100644, size=2456)
Clearing orphaned inode 11535970 (uid=0, gid=0, mode=0100644, size=335240)
Setting free inodes count to 29879660 (was 29737485)
Setting free blocks count to 41417686 (was 20072881)
/dev/mapper/pc--vg-root: clean, 553620/30433280 files, 80298026/121715712 blocks
/dev/mapper/pc--vg-root: status 0, rss 6876, real 38.344677, user 0.482391, sys 0.290328

I don't know how filesystems work, but to my understanding of what I read in the last hours, it looks like fsck just removed the data I was trying to restore? Now extundelete runs without complaints, but

$ sudo extundelete /dev/mapper/pc--vg-root --restore-directory /home/username/path/to/restore
NOTICE: Extended attributes are not restored.
Loading filesystem metadata ... 3715 groups loaded.
Loading journal descriptors ... 0 descriptors loaded.
Searching for recoverable inodes in directory /home/username/path/to/restore...
0 recoverable inodes found.
Looking through the directory structure for deleted files ... 
0 recoverable inodes still lost.
No files were undeleted.

I know I can not restore overwritten data, but I erroneously removed more than 100GB, I don't think they can have been all overwritten before I created the disk image with dd...

Answer 1

Also, sudo fsck -N only wanted to operate on /dev/sdaX.

That doesn't make any sense. e2fsck works on any block device, maybe even on image files.

You should avoid irreversible writes to the image. I suggest you delete the swap volume and use the free space in the VG to create a snapshot of pc--vg-root (lvcreate --snapshot ...). Then you can run e2fsck against the snapshot instead of against the real data. If something goes wrong you can just throw away the snapshot and start over.

I am not familiar with extundelete. A quick search revealed that it is supposed to support ext4 but the message WARNING: EXT3_FEATURE_INCOMPAT_RECOVER is set. sounds like it does not support/like ext4. Is that a very old version of extundelete?

So first run e2fsck, and if the Extended attribute has an invalid value length message remains then try to get a newer version of extundelete.

The whole problem is not related to LUKS and the encryption at all so you might want to change the title of your question.

Answer 2

I managed to restore many, maybe all, of the files I lost with photorec. I read about it in a forum page I can't find anymore and thought "let's just give it a try to this other tool as well".

I ran

$ sudo kpartx -a -v image.iso # map the disk image partitions
add map loop0p1 (254:0): 0 997376 linear 7:0 2048
add map loop0p2 (254:1): 0 2 linear 7:0 1001470
add map loop0p5 (254:2): 0 975769600 linear 7:0 1001472

$ sudo cryptsetup luksOpen /dev/mapper/loop0p5 img # unlock the partition with my data
Enter passhprase for /dev/mapper/loop0p5:

$ lsblk
NAME                  MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
loop0                   7:0    0 465,8G  0 loop
├─loop0p1             254:0    0   487M  0 part
├─loop0p2             254:1    0     1K  0 part
└─loop0p5             254:2    0 465,3G  0 part
  └─img               254:3    0 465,3G  0 crypt
    ├─pc--vg-root   254:4    0 464,3G  0 lvm
    └─pc--vg-swap_1 254:5    0   980M  0 lvm

[...omitting other lsblk output...]

$ sudo vgchange -a y pc-vg
  2 logical volume(s) in volume group "pc-vg" now active

to prepare the partition, then

$ sudo photorec

The first time I got an error about EXT3 I don't remember much about. I tried to run sudo fsck -r /dev/mapper/pc--vg-root and got the same output I wrote in the update to the question. However, after that photorec worked. I don't know exactly what I did but it worked, so I won't comply.

At the second run of photorec I just followed the wizard. I just picked the options I supposed would give me the best result and I don't have the complete history of them, so I will just write what I am sure I did.

• pick the right device (/dev/mapper/pc--vg-root)
• pich the right partition (ext4)
• pick the right partition filesystem ([ ext/ext3 ] ext2/ext3/ext4 filesystem)
• choose to scan the unallocated space only ([ Free ] Scan for file from ext2/ext3 unallocated space only), as I didn't need to restore not deleted files
• selected the position to write restored files to

At some point I also chose the types of file I wanted to restore (text and PDF).

After some hours of analysis and restoring, photorec delivered me some hundreds of subdirectories (recup_dir.<nnn>, where <nnn> is an incremental number) filled with files with random-looking names and correct extension (for example, f582010347.txt is a text file I know I saved with a proper name). I checked some random files and it looks like I got back all the text files I had on the disk, even the undeleted ones (including /etc/ssh/sshd_config, for example), renamed with random-looking names and apparently randomly sorted in subdirectories, and they sum up for more than 80GB in total. But at least I have them back.

In the next days I will try to automatically filter them out to find the ones I need. I did some fast tries and I achieved good results with

$ grep --ignore-case --recursive -B5 -A5 'string' restore_path

where:

• string is a string I know is present in the file I want to restore
• restore_path is the path I instructed photorec to write the recovered file

Adding to --text to grep options I could identify some of the PDF I need to restore as well. However, I know that as PDF are binary files this will probably not allow me to get back all of them.

After all, it went much better than I imagined. The lesson I learned: my regular backups will not protect me against myself. Also, aliasing rm to something less dangerous might be a good idea (this looks interesting).