Understanding iostat with Linux software RAID

Question

I'm trying to understand what I see in iostat, specifically the differences between the output for md and sd devices.

I have a couple of quite large Centos Linux servers, each with E3-1230 CPU, 16 GB RAM and 4 2TB SATA disk drives. Most are JBOD, but one is configure with software RAID 1+0. The servers have very similar type and amount of load, but the %util figures I get with iostat on the software raid one is much higher than others, and I'm trying to understand why. All servers are usually 80-90% idle with regard to CPU.

Example of iostat on a server without RAID:

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           9.26    0.19    1.15    2.55    0.00   86.84

Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
sdb               2.48     9.45   10.45   13.08  1977.55  1494.06   147.50     2.37  100.61   3.86   9.08
sdc               4.38    24.11   13.25   20.69  1526.18  1289.87    82.97     1.40   41.14   3.94  13.36
sdd               0.06     1.28    1.43    2.50   324.67   587.49   232.32     0.45  113.73   2.77   1.09
sda               0.28     1.06    1.33    0.97   100.89    61.63    70.45     0.06   27.14   2.46   0.57
dm-0              0.00     0.00    0.17    0.24     4.49     1.96    15.96     0.01   18.09   3.38   0.14
dm-1              0.00     0.00    0.09    0.12     0.74     0.99     8.00     0.00    4.65   0.36   0.01
dm-2              0.00     0.00    1.49    3.34   324.67   587.49   188.75     0.45   93.64   2.25   1.09
dm-3              0.00     0.00   17.73   42.82  1526.17  1289.87    46.50     0.35    5.72   2.21  13.36
dm-4              0.00     0.00    0.11    0.03     0.88     0.79    12.17     0.00   19.48   0.87   0.01
dm-5              0.00     0.00    0.00    0.00     0.00     0.00     8.00     0.00    1.17   1.17   0.00
dm-6              0.00     0.00   12.87   20.44  1976.66  1493.27   104.17     2.77   83.01   2.73   9.08
dm-7              0.00     0.00    1.36    1.58    95.65    58.68    52.52     0.09   29.20   1.55   0.46

Example of iostat on a server with RAID 1+0:

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           7.55    0.25    1.01    3.35    0.00   87.84

Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
sdb              42.21    31.78   18.47   59.18  8202.18  2040.94   131.91     2.07   26.65   4.02  31.20
sdc              44.93    27.92   18.96   55.88  8570.70  1978.15   140.94     2.21   29.48   4.60  34.45
sdd              45.75    28.69   14.52   55.10  8093.17  1978.16   144.66     0.21    2.95   3.94  27.42
sda              45.05    32.59   18.22   58.37  8471.04  2040.93   137.24     1.57   20.56   5.04  38.59
md1               0.00     0.00   18.17  162.73  3898.45  4013.90    43.74     0.00    0.00   0.00   0.00
md0               0.00     0.00    0.00    0.00     0.00     0.00     4.89     0.00    0.00   0.00   0.00
dm-0              0.00     0.00    0.07    0.26     3.30     2.13    16.85     0.04  135.54  73.73   2.38
dm-1              0.00     0.00    0.25    0.22     2.04     1.79     8.00     0.24  500.99  11.64   0.56
dm-2              0.00     0.00   15.55  150.63  2136.73  1712.31    23.16     1.77   10.66   2.93  48.76
dm-3              0.00     0.00    2.31    2.37  1756.39  2297.67   867.42     2.30  492.30  13.08   6.11

So my questions are:

1) Why is there such a relatively high %util on the server with RAID vs the one without.

2) On the non-RAID server the %util of the combined physical devices (sd*) are more or less the same as the combined LVM devices (dm-*). Why is that not the case for the RAID server?

3) Why does it seem like the software RAID devices (md*) are virtually idle, while the underlying physical devices (sd*) are busy? My first thought was that it might be caused by RAID checking, but /proc/mdadm shows all good.

Edit: Apologies, I thought the question was clear, but that seems there is some confusion about it. Obviously the question is not about the difference in the %util between drives on one server, but why the total/avg %util value on one server is so different from the other. Hope that clarifies any misunderstanding.

Where are you seeing that the raid device is idle? I see `md1` with 162 writes per second. This is more than 4 times as much as all the drives in your other server combined. — phemmer, Jul 10 '14 at 14:09
The only thing that has `%idle` is the cpu, and that's about the same on both hosts. — phemmer, Jul 10 '14 at 15:39
Are you saying the numbers in the %idle category is just for CPU? If so, why is the overall CPU 87% idle? — Dokbua, Jul 10 '14 at 17:47
Are you really asking why there is a ~1% difference between all the counters in both scenario? — Braiam, Jul 10 '14 at 20:47
@Rene if people are misunderstanding you, you might want to consider that you should edit your question and clarify instead of rudely telling them to read again. You ask "Why is there such a relatively high %idle on the server with RAID vs the one without" and the difference between the only idle values (the cpu's) you show is 1%. If that's not what you're asking, then please clarify. The clearer you make this, the likelier it is that someone will answer. For example, why don't you specify exactly which columns confuse you and show us the values instead of expecting us to add them up? — terdon, Jul 11 '14 at 09:58
In scenario 1 the %util value is ~6 avg for the phys devices. In scenario 2 the %util value is ~30 avg for the phys devices. I am asking for the difference between those two scenarios. I think I already made it clear I was asking about the %util columns. I can't see how that can possibly be misunderstood? — Dokbua, Jul 11 '14 at 14:00
@Rene Because you've been mentioning `%idle`, not `%util`. They are completely different statistics. I also explained the difference in the servers, one of your servers is experiencing over 4 times as many writes as the other. — phemmer, Jul 11 '14 at 15:09
@terdon, sorry, my bad, I meant %util, forget about %idle, that is a cpu issue, the question is about disk I/O so it was really just a case of not being able to see the forest for all the trees. I have updated the original question again, but I actually think it was quite clear from my July 11 update. — Dokbua, Jul 25 '14 at 05:04
@Patrick yes that is my question, it seems clear than one server is experiencing 4 times more writes than the other, regardless that the actual load mix, apps, traffic etc is virtually the same. It is for that reason I am wondering if there is some known issue with softraid that cause unnecessary writes — Dokbua, Jul 25 '14 at 05:04

score 1 · Answer 1 · edited Nov 25 '18 at 16:06

non-RAID

Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
dm-3              0.00     0.00   17.73   42.82  1526.17  1289.87    46.50     0.35    5.72   2.21  13.36

RAID

Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
dm-2              0.00     0.00   15.55  150.63  2136.73  1712.31    23.16     1.77   10.66   2.93  48.76

avgrq-sz is lower, w/s is higher. This shows a larger number of smaller I/O requests. Thus the I/O may be more "random". More disk seeks = slower I/O.

Understanding iostat with Linux software RAID

1 Answers1