Disclaimer:
This can be all wrong, I’m on deep water here and it is easy to make a mistake but hard to verify if you are correct. I’m assuming the sectors reported are all 512 bytes. There are also offsets that might be wrong.
I hope I got this correct enough, but time will tell. If you find an error, please do correct me!
All commands need root privileges to run. Backups are always good.
Background
I'm struggling with the same problem as you. Had half of the drives in a RAID6 disappear due to a broken disk controller card. Not good. I now have 48 MD sectors with mismatch and want to find out what files might be stored there. My setup is similar to yours but I also have LVM in the mix.
This answer is for my setup because I think many actually also use LVM and you did not present that many details of your setup. This was also the only one I could test.
In your case just skip the parts around LVM. If you have the md array partitioned you will also have to find the offset to your filesystem. In either case, it should give you some ideas at least.
The Setup
The storage setup looks like this:
Mdraid RAID6 on 6 SATA drives as /dev/md1. This array is used as PV for a LVM2 volume group “vg_arch”. In the VG there are multiple LVs, one of them is “lv_arch2” formatted as Ext4.
So we have: HDD→MD→LVM→Ext4
The problem
After running a check on your mdraid array (/usr/share/mdadm/checkarray -a /dev/md1), you found lines like this in /var/log/syslog:
Jan 4 03:28:28 orion kernel: [36199.659889] md1: mismatch sector in range 2684449552-2684449560
Jan 4 03:28:28 orion kernel: [36199.659897] md1: mismatch sector in range 2684449560-2684449568
Jan 4 03:28:28 orion kernel: [36199.659901] md1: mismatch sector in range 2684449568-2684449576
Jan 4 03:28:28 orion kernel: [36199.659904] md1: mismatch sector in range 2684449576-2684449584
Jan 4 03:28:28 orion kernel: [36199.659921] md1: mismatch sector in range 2684449584-2684449592
Jan 4 03:28:28 orion kernel: [36199.659925] md1: mismatch sector in range 2684449592-2684449600
Problem here is that some blocks in the RAID array are corrupt and the data can not be trusted. The number of different reasons for this is almost unlimited but the sectors are bad and we now have to find out if any files are stored there and if so, what files.
In my case, the array was split 50/50 so it was impossible for mdraid to know what data to use.
The range of bad sectors are 2684449552 – 2684449600, 48 in total.
It is good to first read about RAID recovery and use overlays when trying to recover in order to not destroy your data. It is super easy to destroy it all.
I assume you have your array assembled and working, at least in read-only mode. I used overlays when testing so I did not write anything to the real array by mistake. Only read-only access is needed for this guide.
The sector hunt
To start, the sector number we got from the kernel is the sector of the md array (this is my assumption at least). This is good because we do not need to take the lower levels of the storage stack into consideration (partition offsets etc) and that makes it a bit easier.
This is the path we have to follow: HDD→MD→LVM→Ext4→file
LVM
We are already at the MD level, now we need to look at LVM.
At the bottom of the LVM stack are the physical volumes (PVs). These are split into extents and one or more extents make up a logical volume (LV). The physical volumes are just the underlying device, so in this case that’s /dev/md1.
The data in a PV does not start directly at sector 0 but has an offset. The first sectors are used for LVM metadata. We first have to work out how far into the PV the extents start:
# pvs --noheadings -o pe_start --units s /dev/md1
2048S
The data starts at 2048 sectors in on the PV. The problematic sectors starts at 2684449552-2048 = 2684447504 sectors in LVM and ends at 2684449600-2048 = 335555944.
Next we need to know how big an LVM extent is:
# pvdisplay --units s /dev/md1 | grep 'PE Size'
PE Size 8192 Se
We have 8192 sectors per LVM extent.
Now we can calculate in what extent the problem starts: 2684447504/8192 = 327691,345703125 extents
It is a fractional because the sector is not on an exact PE boundary. The remainder will give us the sector offset into the PE:
0,345703125*8192 = +2832 sectors
Next step is to try to find what LV that is using PE 327691:
# pvdisplay --maps /dev/md1 | grep -A1 'Physical extent'
Physical extent 0 to 2621439:
Logical volume /dev/vg_arch/lv_arch2
--
Physical extent 2621440 to 3801087:
Logical volume /dev/vg_arch/lv_test
We can see that the PE belongs to LV /dev/vg_arch/lv_arch2 and that is has no offset. If it has an offset we have to take that into account.
FS Ext4
We have now enough information that we can go to the filesystem level.
First we have to know what sector/block sizes we are working with:
# fdisk -u -l /dev/md1
Disk /dev/md1: 14.52 TiB, 15959456743424 bytes, 31170813952 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 1048576 bytes / 4194304 bytes
The md array is using 512 byte sectors (I’m a bit unsure if this is correct or not. I’m basically just assuming the kernel uses the same sector size when reporting the error).
To get the Ext4 block size:
# tune2fs -l /dev/vg_arch/lv_arch2 | grep Block
Block count: 2684354560
Block size: 4096
The filesystem uses 4KiB blocks that equates to 8 sectors per block.
Now we have to translate the mdadm sector number to Ext4 block number because they are not the same. To do that we can use this formula:
(mdadm sector number) / ((filesystem block size) / (mdadm sector size)) = Ext4 block number
In this case we have:
2684447504 / (4096 / 512) = 335555938 (start)
2684447552 / (4096 / 512) = 335555944 (stop)
So the problematic block range is 335555938 to 335555944 in the Ext4 filesystem.
File check
When we now have the block numbers we can move on to try finding the file stored there. This can be done using debugfs.
# debugfs
debugfs:
Then in the debugfs console, use the open command to open the filesystem (it must not already be mounted):
debugfs: open /dev/vg_arch/lv_arch2
This operation can take some time if you have a large filesystem.
Now we can start testing if the blocks are used or not. If we are lucky the bad blocks are not used by any file. Use testb blocknumber to test it.
debugfs: testb 335555938
Block 335555938 not in use
Repeat this command for every bad block you have and record any used ones, like this:
debugfs: testb 335555944
Block 335555944 marked in use
If all blocks are unused you are done. Congratulations! If not, then we have to continue to find the files affected.
Next step is to find the inode that is using this block:
debugfs: icheck 335555944
Block Inode number
335555944 279577043
So we have one inode 279577043 that is affected, lets find the filename:
debugfs: ncheck 279577043
Inode Pathname
279577043 /my_files/file.dat
Finally, we have found the file affected by the broken mdadm sectors. That wasn't too hard. ;-)
To end, run the commands close and quit to exit debugfs.
Sources
It was difficult to find information about this problem but here are the sites that I used the most:
