2

I was running a RAID 5 reshape process (growth from 4 to 5 drives) and at night a cooling fan failed and the server suddenly rebooted itself due to CPU overheat and one drive is now possibly out of date.

mdadm --examine /dev/sd[abcdi]

/dev/sda:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x45
     Array UUID : 7b55730e:002db9a2:e79a9acc:ee95da88
           Name : AlienNAS:1
  Creation Time : Mon Jun 14 22:06:14 2021
     Raid Level : raid5
   Raid Devices : 5

 Avail Dev Size : 11720780976 (5588.90 GiB 6001.04 GB)
     Array Size : 23441561600 (22355.62 GiB 24004.16 GB)
  Used Dev Size : 11720780800 (5588.90 GiB 6001.04 GB)
    Data Offset : 264192 sectors
     New Offset : 260096 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : b1af3662:e6d41207:08fd222b:bfe10074

Internal Bitmap : 8 sectors from superblock
  Reshape pos'n : 11643189248 (11103.81 GiB 11922.63 GB)
  Delta Devices : 1 (4->5)

    Update Time : Sun Jan 15 10:09:13 2023
  Bad Block Log : 512 entries available at offset 32 sectors
       Checksum : 16afeaeb - correct
         Events : 22037

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 2
   Array State : AAAA. ('A' == active, '.' == missing, 'R' == replacing)

/dev/sdb:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x45
     Array UUID : 7b55730e:002db9a2:e79a9acc:ee95da88
           Name : AlienNAS:1
  Creation Time : Mon Jun 14 22:06:14 2021
     Raid Level : raid5
   Raid Devices : 5

 Avail Dev Size : 11720780976 (5588.90 GiB 6001.04 GB)
     Array Size : 23441561600 (22355.62 GiB 24004.16 GB)
  Used Dev Size : 11720780800 (5588.90 GiB 6001.04 GB)
    Data Offset : 264192 sectors
     New Offset : 260096 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : dc6f6b93:ca24c965:6773fc72:1b9b36c5

Internal Bitmap : 8 sectors from superblock
  Reshape pos'n : 11646240768 (11106.72 GiB 11925.75 GB)
  Delta Devices : 1 (4->5)

    Update Time : Sun Jan 15 10:10:58 2023
  Bad Block Log : 512 entries available at offset 32 sectors
       Checksum : 40074b23 - correct
         Events : 22042

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 1
   Array State : AAAA. ('A' == active, '.' == missing, 'R' == replacing)

/dev/sdc:

          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x45
     Array UUID : 7b55730e:002db9a2:e79a9acc:ee95da88
           Name : AlienNAS:1
  Creation Time : Mon Jun 14 22:06:14 2021
     Raid Level : raid5
   Raid Devices : 5

 Avail Dev Size : 11720780976 (5588.90 GiB 6001.04 GB)
     Array Size : 23441561600 (22355.62 GiB 24004.16 GB)
  Used Dev Size : 11720780800 (5588.90 GiB 6001.04 GB)
    Data Offset : 264192 sectors
     New Offset : 260096 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : ec41cd88:9f0af810:fb4d44f0:8df26665

Internal Bitmap : 8 sectors from superblock
  Reshape pos'n : 11644702720 (11105.25 GiB 11924.18 GB)
  Delta Devices : 1 (4->5)

    Update Time : Sun Jan 15 10:10:47 2023
  Bad Block Log : 512 entries available at offset 32 sectors
       Checksum : fde14504 - correct
         Events : 22041

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 3
   Array State : AAAA. ('A' == active, '.' == missing, 'R' == replacing)

/dev/sdd:

          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x45
     Array UUID : 7b55730e:002db9a2:e79a9acc:ee95da88
           Name : AlienNAS:1
  Creation Time : Mon Jun 14 22:06:14 2021
     Raid Level : raid5
   Raid Devices : 5

 Avail Dev Size : 11720780976 (5588.90 GiB 6001.04 GB)
     Array Size : 23441561600 (22355.62 GiB 24004.16 GB)
  Used Dev Size : 11720780800 (5588.90 GiB 6001.04 GB)
    Data Offset : 264192 sectors
     New Offset : 260096 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : ff953142:14667e54:d725d785:05b0ee8b

Internal Bitmap : 8 sectors from superblock
  Reshape pos'n : 11644702720 (11105.25 GiB 11924.18 GB)
  Delta Devices : 1 (4->5)

    Update Time : Sun Jan 15 10:10:58 2023
  Bad Block Log : 512 entries available at offset 32 sectors
       Checksum : b6e689e8 - correct
         Events : 22042

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 0
   Array State : AAAA. ('A' == active, '.' == missing, 'R' == replacing)

/dev/sde:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : f9685831:a8225d4c:49ce6aca:2b49612c
           Name : AlienNAS:1
  Creation Time : Thu Jun 17 07:21:06 2021
     Raid Level : raid5
   Raid Devices : 5

 Avail Dev Size : 5860273072 (2794.40 GiB 3000.46 GB)
     Array Size : 11720536064 (11177.57 GiB 12001.83 GB)
  Used Dev Size : 5860268032 (2794.39 GiB 3000.46 GB)
    Data Offset : 260096 sectors
   Super Offset : 8 sectors
   Unused Space : before=260016 sectors, after=5040 sectors
          State : clean
    Device UUID : 76e063b8:ebb80d30:86c69755:629376e0

Internal Bitmap : 8 sectors from superblock
    Update Time : Sun Jan 15 10:59:08 2023
  Bad Block Log : 512 entries available at offset 24 sectors
       Checksum : e93f4335 - correct
         Events : 17214

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 3
   Array State : AAAAA ('A' == active, '.' == missing, 'R' == replacing)

/dev/sdi:

          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x45
     Array UUID : 7b55730e:002db9a2:e79a9acc:ee95da88
           Name : AlienNAS:1
  Creation Time : Mon Jun 14 22:06:14 2021
     Raid Level : raid5
   Raid Devices : 5

 Avail Dev Size : 11720780976 (5588.90 GiB 6001.04 GB)
     Array Size : 23441561600 (22355.62 GiB 24004.16 GB)
  Used Dev Size : 11720780800 (5588.90 GiB 6001.04 GB)
    Data Offset : 264192 sectors
     New Offset : 260096 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : df5806de:0f7ba1b5:4517d85d:96bc4de8

Internal Bitmap : 8 sectors from superblock
  Reshape pos'n : 11646240768 (11106.72 GiB 11925.75 GB)
  Delta Devices : 1 (4->5)

    Update Time : Sun Jan 15 10:10:58 2023
  Bad Block Log : 512 entries available at offset 32 sectors
       Checksum : e86d4fc8 - correct
         Events : 22042

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : spare
   Array State : AAAA. ('A' == active, '.' == missing, 'R' == replacing)

And

mdadm --assemble --force /dev/md126 /dev/sda /dev/sdb /dev/sdc /dev/sdd /dev/sdi --verbose

mdadm: looking for devices for /dev/md126
mdadm: /dev/sda is identified as a member of /dev/md126, slot 2.
mdadm: /dev/sdb is identified as a member of /dev/md126, slot 1.
mdadm: /dev/sdc is identified as a member of /dev/md126, slot 3.
mdadm: /dev/sdd is identified as a member of /dev/md126, slot 0.
mdadm: /dev/sdi is identified as a member of /dev/md126, slot -1.
mdadm: added /dev/sdb to /dev/md126 as 1
mdadm: added /dev/sda to /dev/md126 as 2 (possibly out of date)
mdadm: added /dev/sdc to /dev/md126 as 3
mdadm: no uptodate device for slot 4 of /dev/md126
mdadm: added /dev/sdi to /dev/md126 as -1
mdadm: added /dev/sdd to /dev/md126 as 0
mdadm: /dev/md126 assembled from 3 drives and 1 spare - not enough to start the array.

And

mdadm --detail /dev/md126

/dev/md126:
           Version : 1.2
        Raid Level : raid0
     Total Devices : 5
       Persistence : Superblock is persistent

             State : inactive
   Working Devices : 5

     Delta Devices : 1, (-1->0)
         New Level : raid5
        New Layout : left-symmetric
     New Chunksize : 512K

              Name : AlienNAS:1
              UUID : 7b55730e:002db9a2:e79a9acc:ee95da88
            Events : 22041

    Number   Major   Minor   RaidDevice

       -       8       32        -        /dev/sdc
       -       8        0        -        /dev/sda
       -       8       48        -        /dev/sdd
       -       8       16        -        /dev/sdb
       -       8      128        -        /dev/sdi

And

mdadm --add -v /dev/md126 /dev/sda
mdadm: Cannot get array info for /dev/md126

Any help really appreciated! Thanks.

AMMX
  • 21
  • 2
  • kernel version, mdadm version? in current kernel / mdadm, --examine should not show failed array state or device role spare for a growing raid5 array. so with that drive turned spare, you have one drive missing (no device for slot 4). then for some reason md fails to acknowledge the other 4 drives (out of date despite using --assemble --force). I guess you could try your luck --assemble --force without /dev/sdi as from current state of things, md won't be using that one either way. If data is very important, use overlays for such experiments... – frostschutz Jan 15 '23 at 18:42
  • Reshape pos deviates by ~3GiB, not sure where mdadm would resume by default if you could force reassembling it. All in all this looks like a case where I might resort to hexediting metadata to proceed, or use two sets of overlays, two mdadm --create (reflecting both pre and post grow state) then stitch it with dm-linear at the reshape pos just to get access to files… – frostschutz Jan 15 '23 at 18:47
  • Thanks, mdadm --examine /dev/md126 shows empty output. Kernel is 5.15 and mdadm is 4.1. Force reassemble does not work as stated in my original post :-( – AMMX Jan 15 '23 at 21:01
  • Without sdi: mdadm: /dev/sda is identified as a member of /dev/md126, slot 2. mdadm: /dev/sdb is identified as a member of /dev/md126, slot 1. mdadm: /dev/sdc is identified as a member of /dev/md126, slot 3. mdadm: /dev/sdd is identified as a member of /dev/md126, slot 0. mdadm: added /dev/sdb to /dev/md126 as 1 mdadm: added /dev/sda to /dev/md126 as 2 (possibly out of date) mdadm: added /dev/sdc to /dev/md126 as 3 mdadm: no uptodate device for slot 4 of /dev/md126 mdadm: added /dev/sdd to /dev/md126 as 0 mdadm: /dev/md126 assembled from 3 drives - not enough to start the array. – AMMX Jan 15 '23 at 21:04
  • It seems that `--assemble --force` won't work for your situation, found in mdadm / Assemble.c that drives that report different reshape positions are explicitely skipped. There's this comment: `/* Here, it may be a source of data. If two * devices claim different progresses, it * means that reshape boundaries differ for * their own devices. Kernel will only treat * the first one as reshape progress and * go on. It may cause disaster, so avoid it. */` so you might be stuck here, but it's just a guess from reading the mdadm source code. In that case you can only proceed with manual recovery. – frostschutz Jan 15 '23 at 22:46
  • Thanks again, what would be the best way of manual recovery? Except of modification and recompilation of mdadm to allow reshape with different positions. I run gentoo so I could create a separate modified mdadm package as a last resort. – AMMX Jan 15 '23 at 23:01
  • Full copy of the involved drives or [copy-on-write overlays](https://raid.wiki.kernel.org/index.php/Recovering_a_failed_software_RAID#Making_the_harddisks_read-only_using_an_overlay_file), freeze-reshape, then experiment. Patching mdadm might be an option, unsure what the result might be. Hexediting metadata (same update time, event count, reshape pos for all) is another option, and [mdadm --create](https://unix.stackexchange.com/questions/131915/should-i-use-mdadm-create-to-recover-my-raid) yet another option (only you have to do it twice to reflect the two grow layouts) and stitch at offset. – frostschutz Jan 15 '23 at 23:16
  • …and that's just for the raid recovery, you might then be confronted with inconsistent filesystem issues, depending on how the sudden shutdown / unmount happened. Good luck… – frostschutz Jan 15 '23 at 23:19
  • So I overwrote the information in the RAID superblock, mdadm --examine output is now the same as for other drives with a correct checksum so I am going to re-run the reshape process.... – AMMX Jan 16 '23 at 11:56
  • Reshape is running so I am curious what the result will be :-D – AMMX Jan 16 '23 at 12:02
  • While it is running, I am thinking about what happened. Sounds strange for one drive to end 3GB sooner given the fact the read/write is running simultaneously on all drives and thus possible cached data are interleaved from all drives. Leads me to an idea the superblock on the particular drive was not written yet. In any case tomorrow this time I will see :-D – AMMX Jan 16 '23 at 18:03

0 Answers0