3

I want to exchange a Harddrive which is in a pre-failing state, (some reallocated sectors) which is used in a raid5 array with 3 disks (software raid with mdadm).

Is it possible to set a new harddrive as a hot-spare and initialize a takeover from the failing to the spare drive?

Some methods suggest to add the drive and then set a drive failure command. As I know, in this state the raid5 is degraded and a drive failure would end up in....

so is there a possibility to "copy" the data live (or rebuild the raid) from the failing drive to the spare (without take its parity function away), and as the copy or rebuild process is finished, remove the failing drive.

Anthon
  • 78,313
  • 42
  • 165
  • 222
raoul
  • 31
  • 1

2 Answers2

5

Yes, you can (provided you have a 3.2+ kernel). First, add a new drive as a spare:

mdadm /dev/md0 --add /dev/sdc1

(replace md0 and sdc1 with your RAID and disk device, respectively).

Then, initiate a copy-replace operation like this:

echo want_replacement > /sys/block/md0/md/dev-sdd1/state

Where md0 is, again, your RAID device, and sdd1 is the failing drive. (Actually, sdd1 is a partition on the failing drive -- I prefer to create RAID sets on partitions rather than on raw disks).

The system will copy all readable blocks from sdd1 to sdc1. If it comes to an unreadable block, it will reconstruct it from parity. Once the operation is complete, the former spare (here: sdc1) will become active, and the failing drive will be marked as failed (F) so you can remove it.

Totor
  • 19,302
  • 17
  • 75
  • 102
Ansgar Esztermann
  • 3,205
  • 1
  • 18
  • 21
  • 1
    You learn something new every day. My `mdadm` (3.3) even supports `--replace` so you don't have to muck about with echoing to /sys/. Great answer! – frostschutz Nov 25 '13 at 11:09
0

You could grow it to RAID 6. That would allow for two drives failures. And it would integrate your new disk in your existing RAID without removing the old one.

On the other hand, this will shuffle all data around. So the sync will take longer, and you have to do the same thing again if you want to remove the bad disk and revert to RAID 5. And such grows and shuffling of data comes with its own dangers, particularly in the event of unexpected powerloss. Or in the event of single drive failure, the grow would continue, just without the failed disk, at which point the data of that "failed" disk becomes useless as it no longer shares the same layout with the rest.

Not really recommended overall.

If data safety is your primary concern, (rather than uptime), stick to RAID5 and replace the disk from a rescue system where nothing writes to the RAID while it resyncs; if a drive fails during sync then, you lose nothing that isn't lost already. If you make a backup of each disks md metadata before starting the remove/add/resync, you can easily reassemble using the original configuration.

Most people just readd the drive, and trust it will work; for the case where it does not work, you have to make use of your backups.

It's rare for drives to fail at the same time; in most cases the drive already failed a long time ago, you just never noticed due to lack of monitoring.

frostschutz
  • 47,228
  • 5
  • 112
  • 159
  • *"It's rare for drives to fail at the same time"* Do you have a citation for that? Not arguing, just curious, because I've been hearing a lot of the opposite: the stress placed on the other drives by the resilvering process can very easily cause cascade failures. Even [Wikipedia points toward this being a reason for RAID 6](https://en.wikipedia.org/wiki/RAID#RAID_6). Wikipedia certainly claims it was [the reason for RAID-Z3 in ZFS](https://en.wikipedia.org/wiki/ZFS#Software_RAID_using_ZFS) (RAID-Z3 being three disks' worth of parity, compared to two disks' worth of parity for RAID 6). – user Nov 25 '13 at 15:12
  • Resilvering is linear read/write, no seeks, as such not particularly stressful. And `mdadm` tunes down rates completely when the box has other work to do. Any server that does random I/O stresses its disks more, without them dying particularly often because of it. The theory that disks should suddenly die during rebuilds because of stress just stinks - but that's just my personal experience/opinion, no citation, sorry. If you ran a full read test on your disks recently, I will believe you about simultaneous failure - otherwise I won't and blame you for not testing your drives. – frostschutz Nov 25 '13 at 16:45
  • The most recent full ZFS scrub on my system ran four days ago, does that count? It almost certainly involves a lot more random seeking than a linear read across an entire drive does, too. Good point about linear access; the theory, though, is that if any disk is already *marginal*, which it might be if they are all from the same batch and have seen roughly the same use patterns and environment, it's easy to lose it during the heavy I/O. I hope I won't see that sort of failure to begin with. – user Nov 25 '13 at 18:40