How to create a 3x3TB RAID 5 array without losing data from 2 of the drives?

Question

I currently have 2 3TB HDDs, one that is always almost full, and another with ~200GB free. I would like to purchase an extra 3TB drive and setup a RAID 5 array, but I am concerned about losing the existing data.

I have found that mdadm will be used to create the array, with a command similar to mdadm --create --verbose /dev/md0 --level=5 --raid-devices=2 /dev/sdb2 /dev/sdc2 --spare-devices=1 /dev/sdd2, where /dev/sdb2 and /dev/sdc2 are my existing drives (that have data) and /dev/sdd2 is a new 3TB drive with no data on it. Will this cause me to lose the data on /dev/sdb2 and /dev/sdc2?

My other idea was to somehow create a 2x3TB RAID 5 array without a spare device, where one of the drives in the array is empty and the other has data. Then I could copy my files over from the existing drive to the new (6TB) array, wipe the now redundant drive, and then add it as the spare drive for the array. Although I doubt this would work?

If neither of the above options will work, is there another way to create a RAID 5 array with 2 drives that already have data and 1 that is empty? What about if I were add 2 new empty drives at once, would that open up new options?

I am using Ubuntu Server 16.04.2 with mdadm version 3.3.

Answer 1

You have two 3TB disks storing 6TB data. You want to install one new 3TB disk.

This will allow you to convert the three 3TB disks into a RAID5 array storing 6TB data. However, the process is rather fiddly and the opportunities for losing data somewhere along the route are fairly large.

Steps to achieve the requirement

Let's declare the disks as sda (contains data), sdb (contains data), sdc (new).

1. If at all possible take a backup of all your data, even if you have to borrow a disk from a friend for a few days
2. Create a RAID1 array on the new disk sdc. It should have two members of which one is missing
3. Create a filesystem on this RAID1 array
4. Copy the data from sdb to the new RAID1 array
5. Verify that you have correctly copied the data
6. Add sdb to the RAID1 array
7. Wait for the synchronisation to complete
8. Reboot
9. Grow the RAID1 array to RAID5. It should have three members of which one is missing. to do this you will need 128K of temporary space on an additional disk. You might want to use a USB thumbstick for this. Do not use a RAM disk.
10. Copy the data from sda to the new RAID5 array
11. Verify that you have correctly copied the data
12. Add sda to the RAID5 array
13. Reboot

Worked example

Here is a worked example using three files as disk images.

# Prepare the demonstration
#
dd if=/dev/zero bs=1M count=100 of=sda.img
dd if=/dev/zero bs=1M count=100 of=sdb.img
( echo n; echo p; echo 1; echo; echo; echo w ) | fdisk sda.img  # One primary partition
( echo n; echo p; echo 1; echo; echo; echo w ) | fdisk sdb.img  # One primary partition
losetup --show --find --partscan sda.img
losetup --show --find --partscan sdb.img

# At this point we have /dev/loop0 representing the first disk sda, with /dev/loop0p1
# equivalent to a disk partition sda1. Also /dev/loop1 representing the second disk.

mkfs -t ext4 -L sda /dev/loop0p1
mkfs -t ext4 -L sdb /dev/loop1p1
mkdir -p /mnt/sda1 /mnt/sdb1
mount /dev/loop0p1 /mnt/sda1
mount /dev/loop1p1 /mnt/sdb1
cp -a /usr/local/man/. /mnt/sda1/u.l.man/
mkdir /mnt/sdb1/u.l.etc
cp -a /usr/local/bin/. /mnt/sdb1/u.l.bin/
df -h | grep mnt
umount /mnt/sda1
umount /mnt/sdb1


# Create the third disk
#
dd if=/dev/zero bs=1M count=100 of=sdc.img
( echo n; echo p; echo 1; echo; echo; echo w ) | fdisk sdc.img  # One primary partition
losetup --show --find --partscan sdc.img


# Create the RAID1 array and its filesystem
#
mdadm --create /dev/md1 --level=1 --raid-devices=2 --metadata=default /dev/loop2p1 missing
mkfs -t ext4 -L md1 /dev/md1
mkdir -p /mnt/md1

# Now /dev/loop2 is equivalent to third disk sdc, and /dev/loop2p1 representing sdc1


# Copy the data from sdb to md1
#
mount /dev/loop1p1 /mnt/sdb1
mount /dev/md1 /mnt/md1
cp -a /mnt/sdb1/. /mnt/md1/
umount /mnt/sdb1
umount /mnt/md1


# Complete the RAID1 array
#
mdadm --manage /dev/md1 --add /dev/loop1p1


# Grow the RAID1 array to RAID5
#
mdadm --grow /dev/md1 --level=5 --raid-devices=3 --backup-file=/root/workarea.dat --force
e2fsck -f /dev/md1
resize2fs /dev/md1


# Copy the data from sda to md1
#
mount /dev/loop0p1 /mnt/sda1
mount /dev/md1 /mnt/md1
cp -a /mnt/sda1/. /mnt/md1/
umount /mnt/sda1
umount /mnt/md1


# Add the remaning disk to the RAID5 array
#
mdadm --manage /dev/md1 --add /dev/loop0p1


# All done
#
mdadm --stop /dev/md1
losetup -d /dev/loop0
losetup -d /dev/loop1
losetup -d /dev/loop2
rm sda.img sdb.img sdc.img

You really should ensure you understand the worked example BEFORE touching the live data on your disks. Needless to say, it's your responsibility and I really would recommend a backup before you change your live system.