lost superblock in md raid

Question

Issue on Red Hat Linux 5.

Due to some miscommunication two LUNs in our environment were enlarged from 1.2 TB to 1.7 TB.

Now, after a reboot, mdadm does not find the superblocks to build the array again.

The common format — known as version 0.90 — has a superblock that is 4K long and is written into a 64K aligned block that starts at least 64K and less than 128K from the end of the device (i.e. to get the address of the superblock round the size of the device down to a multiple of 64K and then subtract 64K).

I found some old documentation:

# mdadm -D /dev/md0
/dev/md0:
        Version : 0.90
  Creation Time : Tue Jul 10 17:45:00 2012
     Raid Level : raid1
     Array Size : 1289748416 (1230.00 GiB 1320.70 GB)
  Used Dev Size : 1289748416 (1230.00 GiB 1320.70 GB)
   Raid Devices : 2
  Total Devices : 2
Preferred Minor : 0
    Persistence : Superblock is persistent

    Update Time : Wed Apr 17 15:03:50 2013
         State : active
Active Devices : 2
Working Devices : 2
Failed Devices : 0
  Spare Devices : 0

           UUID : 2799bd51:67eb54d2:1fcd3c90:293311a1
         Events : 0.39

    Number   Major   Minor   RaidDevice State
       0     253       10        0      active sync   /dev/dm-10
       1     253       11        1      active sync   /dev/dm-11

# fdisk -l /dev/dm-10 /dev/dm-11

Disk /dev/dm-10: 1320.7 GB, 1320702443520 bytes
255 heads, 63 sectors/track, 160566 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Disk /dev/dm-10 doesn't contain a valid partition table

Disk /dev/dm-11: 1320.7 GB, 1320702443520 bytes
255 heads, 63 sectors/track, 160566 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Disk /dev/dm-11 doesn't contain a valid partition table

Matthias Urlichs · Answer 1 · 2013-04-21T07:31:04.913

6

Simply run mdadm --build -l1 -n2 /dev/md0 /dev/dm-10 /dev/dm11 to get your data back.

Verify that the devices are correct (or use the aliases from /dev/mapper) before doing this!

Also, if your kernel has already used (or is using) one of these devices, data will be inconsistent. You should set up the second device as a degenerate 1.2 array, copy the data over, start using the new degenerate RAID, and add the first disk to it. If you do not have a current backup, make oneyou know what to do before attempting this.

(Switching to 1.2 is a good idea anyway, as 1.2 RAID superblocks guard against this kind of problem.)

edited Apr 21 '13 at 07:31

answered Apr 17 '13 at 16:34

Matthias Urlichs

280
1
9

This is where I get: "No valid superblock found on /dev/dm-10". – ddeimeke Apr 17 '13 at 20:27
1

Right. Changed `--assemble` to `--build`. – Matthias Urlichs Apr 18 '13 at 07:25

score 4 · Answer 2 · answered Apr 17 '13 at 17:14

If it's RAID 1 with 0.90 superblock, you should be able to mount directly without using the RAID layer at all. Data starts without offset at the beginning of the disk, so there should be no problem. Use the read-only mount option or a read-only loop device for the experiment just in case.

If that works, the simplest method would be to use the other disk to create a new RAID from scratch (preferably 1.2 metadata, which would have avoided this problem and doesn't suffer a 2TB limit), copy the data over, and once that's done, add the disk you copied from to the new RAID.

Yes, I am able to mount. And exactly I will move the data to a new RAID. I think this would be best. — ddeimeke, Apr 17 '13 at 20:28

Stéphane Chazelas · Accepted Answer · 2013-04-18T14:51:02.120

Reverting the devices to their original size should just restore the RAID device.

You can confirm that by doing:

losetup --sizelimit=$((1230*(2**30))) -r /dev/loop1 /dev/dm-10
mdadm -E /dev/loop1

mdadm should now find the superblock if the size is correct. Then you can resize your disks back to that size (-r above is for read-only, so it won't do any harm).

If you actually do want to enlarge the md0 array and keep the 0.9 metadata, one thing you could do is:

dmsetup create d1 --table "0 $((1230*(2**30)/512)) linear /dev/dm-10 0"
dmsetup create d2 --table "0 $((1230*(2**30)/512)) linear /dev/dm-11 0"

Once again,

mdadm -E /dev/mapper/d1

should display the raid device.

Assemble the array on those mapper devices:

mdadm -A /dev/md0 /dev/mapper/d[12]

And then resize the devices to the full extent:

dmsetup suspend d1
dmsetup suspend d2
dmsetup reload d1 --table "0 $(blockdev --getsize /dev/dm-10) linear /dev/dm-10 0"
dmsetup reload d2 --table "0 $(blockdev --getsize /dev/dm-11) linear /dev/dm-11 0"
dmsetup resume d1
dmsetup resume d2

Then, you can use --grow to use that extra space:

mdadm /dev/md0 --grow --size max

Wait for the extra space to be resynced, stop the array, cleanup the extra dm devices and reassemble on the original devices:

mdadm --stop /dev/md0
dmsetup remove d1
dmsetup remove d2
mdadm -A /dev/md0 /dev/dm-1[01]

You can use loop devices to do tests beforehand to verify the procedure. Here's a screen capture of what I did to verify that it worked:

~# truncate -s 1230G a
~# truncate -s 1230G b
~# losetup /dev/loop1 a
~# losetup /dev/loop2 b
~# lsblk /dev/loop[12]
NAME  MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
loop1   7:32   0   1.2T  0 loop
loop2   7:64   0   1.2T  0 loop
~# mdadm --create /dev/md0 --metadata 0.9 --level 1 --raid-devices 2 --assume-clean /dev/loop[12]
mdadm: array /dev/md0 started.
~# lsblk /dev/md0
NAME MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
md0    9:0    0   1.2T  0 raid1
~# truncate -s 1700G a
~# truncate -s 1700G b
~# losetup -c /dev/loop1
~# losetup -c /dev/loop2
~# lsblk /dev/loop[12]
NAME  MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
loop1   7:32   0   1.7T  0 loop
└─md0   9:0    0   1.2T  0 raid1
loop2   7:64   0   1.7T  0 loop
└─md0   9:0    0   1.2T  0 raid1
~# mdadm -E /dev/loop1
mdadm: No md superblock detected on /dev/loop1.
(1)~# mdadm --stop /dev/md0
mdadm: stopped /dev/md0
~# dmsetup create d1 --table "0 $((1230*(2**30)/512)) linear /dev/loop1 0"
~# dmsetup create d2 --table "0 $((1230*(2**30)/512)) linear /dev/loop2 0"
~# mdadm -A /dev/md0 /dev/mapper/d[12]
mdadm: /dev/md0 has been started with 2 drives.
~# lsblk /dev/mapper/d[12]
NAME       MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
d1 (dm-19) 253:19   0   1.2T  0 dm
└─md0        9:0    0   1.2T  0 raid1
d2 (dm-20) 253:20   0   1.2T  0 dm
└─md0        9:0    0   1.2T  0 raid1
~# dmsetup suspend d1
~# dmsetup suspend d2
~# dmsetup reload d1 --table "0 $(blockdev --getsize /dev/loop1) linear /dev/loop1 0"
~# dmsetup reload d2 --table "0 $(blockdev --getsize /dev/loop2) linear /dev/loop2 0"
~# dmsetup resume d1
~# dmsetup resume d2
~# lsblk /dev/mapper/d[12]
NAME       MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
d1 (dm-19) 253:19   0   1.7T  0 dm
└─md0        9:0    0   1.2T  0 raid1
d2 (dm-20) 253:20   0   1.7T  0 dm
└─md0        9:0    0   1.2T  0 raid1
~# mdadm /dev/md0 --grow --assume-clean --size max
mdadm: component size of /dev/md0 has been set to 1782579136K
~# lsblk /dev/md0
NAME MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
md0    9:0    0   1.7T  0 raid1
~# cat /proc/mdstat
Personalities : [raid1]
md0 : active raid1 dm-19[0] dm-20[1]
      1782579136 blocks [2/2] [UU]

unused devices: <none>
~# mdadm --stop /dev/md0
mdadm: stopped /dev/md0
~# dmsetup remove d1
~# dmsetup remove d2
~# mdadm -A /dev/md0 /dev/loop[12]
mdadm: /dev/md0 has been started with 2 drives.
~# lsblk /dev/md0
NAME MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
md0    9:0    0   1.7T  0 raid1
~# uname -rs
Linux 3.7-trunk-amd64

You actually can't grow arrays that store the metadata at the end; the kernel won't move the superblock. — psusi, Apr 17 '13 at 23:21
@psusi, I did test that procedure on a raid1 on loop devices and it did grow the array. — Stéphane Chazelas, Apr 18 '13 at 06:36
Using metadata format 0.9? Last time I tried it wouldn't work; I had to rebuild the array using 1.2. Discussion on the linux-raid mailing list seemed to indicate this was to be expected. — psusi, Apr 18 '13 at 13:40
@psusi. I've added a screen capture of a run at the procedure. That's on Linux 3.7. — Stéphane Chazelas, Apr 18 '13 at 14:53

lost superblock in md raid

3 Answers3