3

I have a Hydra LCM RAID device with 4 Bays (4x 2TB Hitachi). I was running it in RAID5 mode since 2010. Last year in May the device was starting giving messages that one drive was degraded and I should replace it. So did I. One month later the second drive was issuing degraded messages. So I also replaced this one.

After the second drive was successfully restored, everything seemed to be working fine. Then some days later, when I started the storage box again, suddenly, it did not detect the RAID mode anymore. The display signaled I have not initialized any disks or mode.

I'm really frustrated since the device is discontinued (since 2015 I think). I'm in the hope that the manufacturer was using a "standard RAID technique" that I might be able to restore this Hardware RAID with some Software RAID alternative (e.g. mdadm).

In the hope this is helpful:

The RAID controller inside the Hydra Super-S LCM is using a backward parity rotation and the RAID stripes are 512 sectors, so all disks are accessed in a balanced manner and the parity disk has no additional workload.

enter image description here

Does anyone know if there is a chance to restore this specific Hardware RAID5 using mdadm or something similar?

Btw. An additional challenge might be that the disks are formatted in some OSX filesystem. Still, I have some USB3 disk reader ready which is currently attached to my Ubuntu. This adapter is able to connect all 4 drives at once. I'm just afraid to run anything like mdadm in the fear it overwrite any existing file system tables or RAID information (or what is left of it). Any tips are highly appreciated.

david
  • 131
  • 5

1 Answers1

4

Make sure to run your experiments in read-only mode:

A naive attempt at re-creating your RAID layout:

# mdadm --create /dev/md100 --assume-clean --metadata=0.90 --level=5 --chunk 256K --raid-devices=4 /dev/loop[0123]

Overwriting it with trace data (data = offset in hex):

# for ((i=0; 1; i+=16)); do printf "%015x\n" $i; done > /dev/md100
# hexdump -C /dev/md100
00000000  30 30 30 30 30 30 30 30  30 30 30 30 30 30 30 0a  |000000000000000.|
00000010  30 30 30 30 30 30 30 30  30 30 30 30 30 31 30 0a  |000000000000010.|
00000020  30 30 30 30 30 30 30 30  30 30 30 30 30 32 30 0a  |000000000000020.|
00000030  30 30 30 30 30 30 30 30  30 30 30 30 30 33 30 0a  |000000000000030.|

In this layout, where are blocks located?

# grep -ano $(printf "%015x" $((0 * 512*512))) /dev/loop[0123]
/dev/loop0:1:000000000000000 # Disk A 1
# grep -ano $(printf "%015x" $((1 * 512*512))) /dev/loop[0123]
/dev/loop1:1:000000000040000 # Disk B 2
# grep -ano $(printf "%015x" $((2 * 512*512))) /dev/loop[0123]
/dev/loop2:1:000000000080000 # Disk C 3
# grep -ano $(printf "%015x" $((3 * 512*512))) /dev/loop[0123]
/dev/loop3:16385:0000000000c0000 # Disk D 4
# grep -ano $(printf "%015x" $((4 * 512*512))) /dev/loop[0123]
/dev/loop0:16385:000000000100000 # Disk A 5

So this is close, but not exactly as shown in your picture. That's the issue with RAID layouts, it might be similar enough, it might even mount, but then show weird corruption in files, since just a few chunks end up being out of order.

With mdadm, the default 4-disk RAID5 layout left-symmetric, if you read the first 4 blocks, it actually reads them from 4 disks. In your illustrated layout, it would read from 3 disks since block 4 is again on the first disk instead of the fourth.

So, to match your picture, you have to try another layout.

Let's go with left-asymmetric.

# mdadm --create /dev/md100 --assume-clean --metadata=0.90 --level=5 --layout=left-asymmetric --chunk 256K --raid-devices=4 /dev/loop[0123]
# for ((i=0; 1; i+=16)); do printf "%015x\n" $i; done > /dev/md100
# mdadm --stop /dev/md100
# echo 3 > /proc/sys/vm/drop_caches
# for i in {0..23}; do grep -ano $(printf "%015x" $(($i * 512*512))) /dev/loop[0123]; done

Output (comments added for better understanding):

/dev/loop0:1:000000000000000 # Disk A 1
/dev/loop1:1:000000000040000 # Disk B 2
/dev/loop2:1:000000000080000 # Disk C 3
# skips parity loop3
/dev/loop0:16385:0000000000c0000 # Disk A 4
/dev/loop1:16385:000000000100000 # Disk B 5
# skips parity loop2
/dev/loop3:16385:000000000140000 # Disk D 6
/dev/loop0:32769:000000000180000 # Disk A 7
# skips parity loop1
/dev/loop2:32769:0000000001c0000 # Disk C 8
/dev/loop3:32769:000000000200000 # Disk D 9
# skips parity loop0
/dev/loop1:49153:000000000240000 # Disk B 10
/dev/loop2:49153:000000000280000 # Disk C 11
/dev/loop3:49153:0000000002c0000 # Disk D 12
/dev/loop0:65537:000000000300000 # Disk A 13
/dev/loop1:65537:000000000340000 # Disk B 14
/dev/loop2:65537:000000000380000 # Disc C 15
# skips parity loop3
/dev/loop0:81921:0000000003c0000 # Disk A 16
/dev/loop1:81921:000000000400000 # Disk B 17
# skips parity loop2
/dev/loop3:81921:000000000440000 # Disk D 18
/dev/loop0:98305:000000000480000 # Disk A 19
# skips parity loop1
/dev/loop2:98305:0000000004c0000 # Disk C 20
/dev/loop3:98305:000000000500000 # Disk D 21
# skips parity loop0
/dev/loop1:114689:000000000540000 # Disk B 22
/dev/loop2:114689:000000000580000 # Disk C 23
/dev/loop3:114689:0000000005c0000 # Disk D 24

This layout seems to match your picture much better. Maybe, it will work. Good luck.

frostschutz
  • 47,228
  • 5
  • 112
  • 159
  • Thank you for the very elaborative answer! All of this is new to me, but I will give it a try and keep you updated. – david Jul 10 '20 at 07:57
  • 1
    @david added another link to one of my earlier answers regarding mdadm --create in general ... you must know the correct drive order, chunk size, layout, etc. and if you don't know, determine by trial & error or deduce it from on-disk data. good luck – frostschutz Jul 10 '20 at 08:14
  • I've created 4 loop devices. Now, when I'm trying to assemble them with mdadm each device raises an error "mdadm: super0.90 cannot open /dev/loop1: Device or resource busy mdadm: /dev/loop1 is not suitable for this array." – david Jul 10 '20 at 08:40
  • Please be careful. I'm using loop devices just to experiment with the RAID layout. If you're doing the overlays as linked in the raid wiki, you'll be using /dev/mapper devices instead. – frostschutz Jul 10 '20 at 08:45
  • Ok, now I got it. I'll be using the devices in /dev/mapper instead. – david Jul 10 '20 at 09:05
  • Now, I'm really not sure to move on. There is a partition table on drive 4 - is this normal? "sudo mdadm --create /dev/md100 --assume-clean --metadata=0.90 --level=5 --layout=left-asymmetric --chunk 256K --raid-devices=4 /dev/mapper/sdb /dev/mapper/sdc /dev/mapper/sdd /dev/mapper/sde mdadm: partition table exists on /dev/mapper/sde Continue creating array?" – david Jul 10 '20 at 09:24
  • @david do you have the correct drive order? in a hardware raid with metadata at the end (blind assumption by me as that's very common for hardware raid), you'd expect to find the partition table on the first disk, not the last. in any case, with the overlays you can try whatever and see what comes out. – frostschutz Jul 10 '20 at 09:31
  • Yes, the order should be correct (I've labled the disks and turned them on in the right order). Just in case - if there is something gone, I should just remove the overlays and re-create them? – david Jul 10 '20 at 09:35
  • Since the partition table seems to be at the last disk, does this have any implications on the mdadm arguments? – david Jul 10 '20 at 09:37
  • 1
    You'll just have to go through some trial & error there. It's also normal for the partition table to be on both 1st and 4th drive (duplicated in parity) if the first chunk on 2nd and 3rd drive is all zero (which might be the case if first partition starts at 1MiB so block 1 has partition table, block 2, 3 are zero, so parity is a duplication of partition table). – frostschutz Jul 10 '20 at 09:40
  • The array ist started ("mdadm: array /dev/md100 started."). What to do next? – david Jul 10 '20 at 09:59
  • Okay, to my understanding I need to run your test-output for "left-asymmetric" on my /dev/mapper devices now, and see if there are similarieties in the output. Which I'm doing now. The first output is "/dev/mapper/sdb:843633:000000000000000". Nothign else yet. Guess this will take some time. – david Jul 10 '20 at 10:22
  • @david my output is just to demonstrate the layout itself. I could simply have stated "your picture looks like left-asymmetric layout" without further explanations and been done with that, but I wanted to show how to come to that conclusion without relying on external sources. Sorry if that part confused you; the only interesting bit for you is the mdadm create command itself, none of the other commands matter. – frostschutz Jul 10 '20 at 11:00
  • Let us [continue this discussion in chat](https://chat.stackexchange.com/rooms/110447/discussion-between-david-and-frostschutz). – david Jul 10 '20 at 11:05