3

I am doing some experiments with hybrid RAID in Linux. My test consists of the following:

2x256GB SSD in RAID 0 (/dev/md1)

2x256GB HDD in RAID 0 (/dev/md2)

Then I made md1 and md2 into a RAID 1 (/dev/md127) and marking the slow HDD (md2) as --write-mostly.

Essentially, my goal is to get maximum performance AND disk space out of my SSDs, but at the same time be "safe" from drive failures. I understand that losing one of the SSDs would mean that I fall back on slow HDDs, but that's a price I am willing to pay compared to losing all data. Besides, it would only be for a few hours until the broken SSDs gets replaced and RAID repaired.

root@s1 / # cat /proc/mdstat
Personalities : [raid0] [raid1] [linear] [multipath] [raid6] [raid5] [raid4] [raid10]

md2 : active raid0 sdd1[1] sdc1[0]
      498802688 blocks super 1.2 512k chunks

md127 : active raid1 md1[2] md2[1](W)
      498671616 blocks super 1.2 [2/2] [UU]
      bitmap: 1/4 pages [4KB], 65536KB chunk

md1 : active raid0 sdb2[1] sda2[0]
      498802688 blocks super 1.2 512k chunks

Now, running a simple throughput benchmark on the 3 raid devices gives a (for me) surprising results:

root@s1 / # hdparm -t /dev/md1

/dev/md1:
 Timing buffered disk reads: 2612 MB in  3.00 seconds = 870.36 MB/sec
root@s1 / # hdparm -t /dev/md2

/dev/md2:
 Timing buffered disk reads: 812 MB in  3.01 seconds = 270.14 MB/sec
root@s1 / # hdparm -t /dev/md127

/dev/md127:
 Timing buffered disk reads: 1312 MB in  3.00 seconds = 437.33 MB/sec

RAID 0 SSD gives 870 MB/sec

RAID 0 HDD gives 270 MB/sec

RAID 1 HYBRID gives 437 MB/sec.

As the HDD raid has been marked as --write-mostly, I would assume that a pure read test would not touch the HDD at all, so what is going on here? I would assume that the hybrid benchmark would give similar results as the pure RAID 0 SSD.

At a first glance, it looks like the HDD somehow is slowing down the RAID, by being partly used for the read (even though I told it not to do reads on the HDD). However, if I have a file copy running on the HDDs while running the hdparm benchmark, I get the same result! If the HDDs WERE used, I would assume the benchmark would give even slower results if the HDDs were used for other tasks during the benchmark.

Jeff Schaller
  • 66,199
  • 35
  • 114
  • 250
Daniele Testa
  • 203
  • 1
  • 7
  • The great difference in speed between the SSD and HDD could indeed be problematic. – K7AAY May 21 '19 at 17:27
  • 1
    for the glory of science, what happens if you fail the hdd side from the RAID1? You could also try RAID1 below and RAID0 on top. This would also mean rewriting only 1 drive in failure case... – frostschutz May 21 '19 at 17:51
  • Note, you might also look at [bcache](https://wiki.archlinux.org/index.php/Bcache) instead of raid as a solution. See questions with the tag [tag:bcache] – meuh May 22 '19 at 09:10
  • Thanks! I will look into bcache. – Daniele Testa May 22 '19 at 10:51
  • The [raid wiki](https://raid.wiki.kernel.org/index.php/Write-mostly) says *Read performance will be somewhere between the fastest device and the slowest device. This is because md WILL occasionally still attempt to read from the slower device* – meuh May 22 '19 at 12:56
  • @meuh Yes, I am aware of this. But IF the HDD was touched, the result should have been affected when I copy a file at the same time on the HDD. Anyways, I did a benchmark with "fio" and it shows 180k IOPS when doing 4kb random reads, so I am pretty sure it never touches the HDD in that case :) – Daniele Testa May 22 '19 at 14:01
  • Another toolset you might try is `blktrace` as I mention [here](https://unix.stackexchange.com/a/402615/119298). I don't know how it would work for a raid setup though. – meuh May 22 '19 at 14:07

0 Answers0