2

I've got a RAID mirror (md1) with three devices (sdc, sdd, sdf). Something went wrong overnight, the array went readonly, and fsck found thousands of inodes with inline_data_fl set and/or on the orphaned inode list.

I've stopped the array with mdadm --manage /dev/md1 --stop, and am running smartctl -t long on each device.

I presume there's a reasonable chance that, when smartctl finished, the devices will differ and thus the array won't reassemble. Is there a standard(?) way of diff'ing three filesystems? And/or doing some sort of automated best-of-three error correction?

I'm running Debian Stable, and have on-disk and off-site backups. I also have a vague plan of shifting to ZFS, to avoid exactly this scenario ...

Kern.log:

Jul 26 15:15:16 jumbo kernel: [1839678.260050] md: md1: data-check done.
Aug  1 01:37:02 jumbo kernel: [2308990.999405] EXT4-fs error (device md1): ext4_lookup:1591: inode #179405745: comm rsync: iget: bad extra_isize 9234 (inode size 256)
Aug  1 01:37:02 jumbo kernel: [2308991.116973] Aborting journal on device md1-8.
Aug  1 01:37:02 jumbo kernel: [2308991.146777] EXT4-fs (md1): Remounting filesystem read-only
Aug  1 01:37:02 jumbo kernel: [2308991.146903] EXT4-fs error (device md1): ext4_lookup:1591: inode #179405746: comm rsync: iget: bad extra_isize 28338 (inode size 256)
Aug  1 01:37:02 jumbo kernel: [2308991.183721] EXT4-fs error (device md1): ext4_lookup:1591: inode #179405747: comm rsync: iget: bad extra_isize 9234 (inode size 256)
Aug  1 01:37:02 jumbo kernel: [2308991.228169] EXT4-fs error (device md1): ext4_lookup:1591: inode #179405748: comm rsync: iget: bad extra_isize 28338 (inode size 256)
Aug  1 01:37:02 jumbo kernel: [2308991.272639] EXT4-fs error (device md1): ext4_lookup:1591: inode #179405749: comm rsync: iget: bad extra_isize 9234 (inode size 256)
Steve Almond
  • 166
  • 6
  • Do you have any log messages of the failure event? `smartctl -t` does not change data. There is no automated best-of-three either. If you already ran fsck on the RAID, unless it's degraded (kicked a disk) it's on all disks anyway. RAID isn't a backup. – frostschutz Aug 01 '20 at 12:22
  • 1
    I have daily, weekly, monthly backups on-disk, on-site, and off-site – Steve Almond Aug 01 '20 at 19:40
  • 1
    Your "RAID mirror" is RAID 1 across three devices, yes? What do the RAID superblocks tell you on each of the `/dev/sd?` devices (`mdadm --examine /dev/sd{c,d,f}`)? You _may_ be able to mount the three separate filesystems as read-only (mind the journals!) and then `diff` them – roaima Aug 01 '20 at 19:41

0 Answers0