Brief context
I have a QNAP TS-451. The NAS has failed, but I'm pretty sure the drives are still fine. I have a new QNAP TS-462 to try to replace it and I'd like to ensure my data is intact in the new system.
Question
How can I access the contents of the LVM when I cannot assemble the RAID?
(more details to the question below)
Environment
The TS-451 system is dead and no longer boots. It was configured like so:
- Drive A - 20TB RAID1 mirror
- Drive B - 20TB RAID1 mirror
- Drive C - 8TB no RAID
- Drive D - 14TB no RAID
The TS-462 is a new system that I want to migrate Drives A/B/C/D to.
Separate Linux system to (ideally) non-destructively read data from drives A/B/C/D:
- Fedora 38
- Linux kernel 6.2.14-300
- Old BIOS (i.e. I do not think it can boot off GPT partitions -- not a problem in this case)
- Pentium Dual Core E6500 (just to give you an idea how old this system is)
Experiment 1
I tried moving one of the drives that is not important (Drive C) to the new TS-462, but it appears that the TS-462 is not able to read it. From what I can tell, it seems confused about the partition table (fdisk reports one thing and blkid/lsblk reports differently.
On a separate Linux computer (Fedora 38), I'm able to read the LVM and mount the partition off Drives C & D and confirm the files are intact.
Experiment 2
So I tried to do the same thing to read off Drive B. Drives A/B are part of a RAID1 mirror, so I figured it should be fine to just experiment (cautiously) on one of the drives and leave the other as backup.
Unfortunately, I don't seem to be able to activate the LVM. Here are some of the things I've tried on the Linux system:
# fdisk -l /dev/sdc
Disk /dev/sdc: 18.19 TiB, 20000588955648 bytes, 39063650304 sectors
Disk model: WDC WD201KFGX-68
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: DFC9F3CE-9CFA-4251-8DD1-48BF99476C04
Device Start End Sectors Size Type
/dev/sdc1 40 1060289 1060250 517.7M Microsoft basic data
/dev/sdc2 1060296 2120579 1060284 517.7M Microsoft basic data
/dev/sdc3 2120584 39045853979 39043733396 18.2T Microsoft basic data
/dev/sdc4 39045853984 39046914269 1060286 517.7M Microsoft basic data
/dev/sdc5 39046914272 39063621869 16707598 8G Microsoft basic data
From what I gather from http://www.rodsbooks.com/linux-fs-code/, the fact the these partitions show up as Microsoft basic data is not a problem, but just evidence that this was setup on a very old version of Linux/fdisk. (I've had the TS-451 since about 2015).
# gdisk /dev/sdc
GPT fdisk (gdisk) version 1.0.9
Partition table scan:
MBR: protective
BSD: not present
APM: not present
GPT: present
Found valid GPT with protective MBR; using GPT.
Command (? for help): p
Disk /dev/sdc: 39063650304 sectors, 18.2 TiB
Model: WDC WD201KFGX-68
Sector size (logical/physical): 512/4096 bytes
Disk identifier (GUID): DFC9F3CE-9CFA-4251-8DD1-48BF99476C04
Partition table holds up to 128 entries
Main partition table begins at sector 2 and ends at sector 33
First usable sector is 34, last usable sector is 39063650270
Partitions will be aligned on 8-sector boundaries
Total free space is 28423 sectors (13.9 MiB)
Number Start (sector) End (sector) Size Code Name
1 40 1060289 517.7 MiB 0700 primary
2 1060296 2120579 517.7 MiB 0700 primary
3 2120584 39045853979 18.2 TiB 0700 primary
4 39045853984 39046914269 517.7 MiB 0700 primary
5 39046914272 39063621869 8.0 GiB 0700 primary
# cat /proc/mdstat
Personalities :
md123 : inactive sdc5[1](S)
8353780 blocks super 1.0
md124 : inactive sdc4[25](S)
530124 blocks super 1.0
md125 : inactive sdc1[26](S)
530108 blocks super 1.0
md126 : inactive sdc2[1](S)
530124 blocks super 1.0
md127 : inactive sdc3[2](S)
19521865728 blocks super 1.0
unused devices: <none>
# mdadm --assemble --readonly /dev/sdc3
mdadm: device /dev/sdc3 exists but is not an md array.
Why??? :-(
pvdisplay and the various LVM tools did not discover the LVMs initially. And when specifying the partition explicitly:
# pvdisplay /dev/sdc3
Cannot use /dev/sdc3: device is an md component
What? But mdadm said it wasn't. :-(
# mdadm --examine /dev/sdc3
/dev/sdc3:
Magic : a92b4efc
Version : 1.0
Feature Map : 0x0
Array UUID : 5a6e38ee:eb6bf302:79856803:58887046
Name : 1
Creation Time : Wed Nov 25 03:18:54 2015
Raid Level : raid1
Raid Devices : 2
Avail Dev Size : 39043731456 sectors (18.18 TiB 19.99 TB)
Array Size : 19521865728 KiB (18.18 TiB 19.99 TB)
Super Offset : 39043733376 sectors
Unused Space : before=0 sectors, after=1920 sectors
State : active
Device UUID : f2a96ebc:996e4231:1576a39a:8606a71c
Update Time : Sun Mar 26 00:56:13 2023
Checksum : 893841dd - correct
Events : 436627
Device Role : Active device 1
Array State : AA ('A' == active, '.' == missing, 'R' == replacing)
Checksum is good. So shouldn't that mean this is a valid RAID partition? (Although the other partitions (even the swap partitions) show valid raid info and checksums, so I don't know what that means.)
Let's try a different route...
However, by setting md_component_detection=0 in /etc/lvm/lvm.conf, and doing pvscan --cache --devices /dev/sdc, I was able to get at the LVM data...
# pvdisplay /dev/sdc3
--- Physical volume ---
PV Name /dev/sdc3
VG Name vg1
PV Size 18.18 TiB / not usable 0
Allocatable yes (but full)
PE Size 4.00 MiB
Total PE 4766080
Free PE 0
Allocated PE 4766080
PV UUID Xt2uVv-PMCy-oHuK-0UAc-43ZI-z7TH-hHAK7A
# vgdisplay vg1
--- Volume group ---
VG Name vg1
System ID
Format lvm2
Metadata Areas 1
Metadata Sequence No 131
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 2
Open LV 0
Max PV 0
Cur PV 1
Act PV 1
VG Size 18.18 TiB
PE Size 4.00 MiB
Total PE 4766080
Alloc PE / Size 4766080 / 18.18 TiB
Free PE / Size 0 / 0
VG UUID oNdjeV-lPuv-PgPR-J11o-zbHQ-X7az-93sr9J
# lvdisplay vg1
--- Logical volume ---
LV Path /dev/vg1/lv544
LV Name lv544
VG Name vg1
LV UUID z1gyiX-FhGG-cTaM-shPx-wZg4-G1xN-b9fS37
LV Write Access read/write
LV Creation host, time NASEFF3AE, 2015-11-25 03:18:56 -0800
LV Status NOT available
LV Size 144.00 GiB
Current LE 36864
Segments 2
Allocation inherit
Read ahead sectors 8192
--- Segments ---
Logical extents 0 to 5119:
Type linear
Physical volume /dev/sdc3
Physical extents 0 to 5119
Logical extents 5120 to 36863:
Type linear
Physical volume /dev/sdc3
Physical extents 1428360 to 1460103
--- Logical volume ---
LV Path /dev/vg1/lv1
LV Name lv1
VG Name vg1
LV UUID 4lsaNW-E4Bg-g93F-ko0a-Ep1m-wHD0-3Hc3Cb
LV Write Access read/write
LV Creation host, time NASEFF3AE, 2015-11-25 03:19:03 -0800
LV Status NOT available
LV Size 18.04 TiB
Current LE 4729216
Segments 2
Allocation inherit
Read ahead sectors 8192
--- Segments ---
Logical extents 0 to 1423239:
Type linear
Physical volume /dev/sdc3
Physical extents 5120 to 1428359
Logical extents 1423240 to 4729215:
Type linear
Physical volume /dev/sdc3
Physical extents 1460104 to 4766079
Yay, so I should be able to mount the drives, right?
# vgchange -ay vg1
WARNING: PV /dev/sdc3 in VG vg1 is using an old PV header, modify the VG to update.
device-mapper: reload ioctl on (253:3) failed: Device or resource busy
device-mapper: reload ioctl on (253:3) failed: Device or resource busy
0 logical volume(s) in volume group "vg1" now active
Hrm... so maybe md_component_detection=0 is not the right approach?
Experiment 3
Just for thoroughness, I tested foremost, scalpel, and testdisk. These are fine tools, but I feel a bit cumbersome for the current situation. The disks should be perfectly fine and readable... partitions and filesystems intact. I'm assuming there's just some sort of version incompatibility somewhere?
Goal & Question (revisited)
My goal is primarily to access my old data (from Drive A/B) one way or another. I don't mind reformatting one of the drives and migrating from another system. Or putting it in the TS-462 if I had any reasonable expectation of being able to access the data.
In my current path (along Experiment 2), where I am stuck is figuring out how to get Linux to recognize the RAID? I think I'm going to try frostschutz's excellent suggestion of copy-on-write overlays shared in Recover files from Linux Raid1 member disk - as bad as it gets.
But I'm open to suggestions!
