How to detect if a disk is seeking?

Question

I can use a variety of tools to measure the volume of disk I/O currently flowing through the system (such as iotop and iostat) but I'm curious if it's possible to easily detect if a disk is seeking a lot with only a small amount of I/O.

I know it;s possible to extract this information using blktrace and then decode it using btt but these are somewhat unwieldy and I was hoping there was a simpler alternative?

Shouldn't a low ratio `(rkB/s + wkB/s)/%util` signify that the disk is seeking (in `iostat -x`)? — Marco, Aug 23 '13 at 14:38
Do you want a tool that shows this live or are you looking to look at the data over a given period of time? — slm, Aug 23 '13 at 15:08
AFAIK, https://www.kernel.org/doc/Documentation/iostats.txt are the only stats maintained by the kernel, so I don't expect you'll find better than blktrace. — Stéphane Chazelas, Aug 23 '13 at 15:22

score 8 · Answer 1 · edited Apr 13 '17 at 12:13

Not sure if you're looking for tools that show this type of information in a real-time type of way or over a period of time, but here are 2 tools that show the real-time aspects of the disk being accessed.

nmon

You invoke it like so, nmon. Then once it's open you hit the j (Filesystems) followed by a d (Disk I/O Graphs D=Stats). See the built-in help (h) for more details.

$ nmon
┌nmon─13g─────────────────────Hostname=manny────────Refresh= 2secs ───11:15.32─────────────────────────────────────────────────────┐
│ Filesystems ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────│
│Filesystem            SizeMB  FreeMB %Used Type     MountPoint                                                                    │
│v/mapper/ubuntu-root 465389.0 457483.9   1.7 ext4     /                                                                           │
│/proc                                      proc     not a real filesystem                                                         │
│/sys                                       sysfs    not a real filesystem                                                         │
│/sys/fs/fuse/connections                   fusectl  not a real filesystem                                                         │
│/sys/kernel/debug                          debugfs  not mounted                                                                   │
│/sys/kernel/security                       security not a real filesystem                                                         │
│/dev                                       devtmpfs not a real filesystem                                                         │
│/dev/pts                                   devpts   not a real filesystem                                                         │
│tmpfs                  740.5   739.4   0.1 tmpfs    /run                                                                          │
│none                     5.0     5.0   0.0 tmpfs    /run/lock                                                                     │
│none                  1851.2  1849.8   0.1 tmpfs    /run/shm                                                                      │
│none                   100.0    99.9   0.1 tmpfs    /run/user                                                                     │
│/dev/sda1              910.9   606.5  33.4 ext2     /boot                                                                         │
│/run/rpc_pipefs                            rpc_pipe fstatfs returned zero blocks!!                                                │
│/run/user/emma/gvfs                        fuse.gvf not mounted                                                                   │
│/run/user/emily/gvfs                       fuse.gvf not mounted                                                                   │
│ Disk I/O ──/proc/diskstats────mostly in KB/s─────Warning:contains duplicates─────────────────────────────────────────────────────│
│DiskName Busy  Read WriteMB|0          |25         |50          |75       100|                                                    │
│sda      100%    0.0   66.4|RWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW>                                                    │
│sda1       0%    0.0    0.0|>                                                |                                                    │
│sda2       0%    0.0    0.0|>                                                |                                                    │
│sda5     100%    0.0   66.4|RWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW>                                                    │
│dm-0     100%    0.0   48.0|RWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW>                                                    │
│dm-1       0%    0.0    0.0|                     >                           |                                                    │
│Totals Read-MB/s=0.0      Writes-MB/s=180.9    Transfers/sec=363.2                                                                │
│──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────│

Another tool I've come across that does at least a live view of disk I/O is atop.

$ atop
ATOP - grinchy                           2013/08/23  11:10:58                           ------                           10s elapsed
PRC | sys    2.18s  | user  26.26s  | #proc    315  | #trun  5 |  #tslpi   764 |  #tslpu     1 |  #zombie    0 |  #exit  1 |
CPU | sys      22%  | user    264%  | irq       0%  | idle    110% |  wait      4% |  guest     0% |  curf 1.20GHz |  curscal  44% |
cpu | sys       5%  | user     76%  | irq       0%  | idle     19% |  cpu003 w  0% |  guest     0% |  curf 1.20GHz |  curscal  44% |
cpu | sys       5%  | user     67%  | irq       0%  | idle     26% |  cpu001 w  2% |  guest     0% |  curf 1.20GHz |  curscal  44% |
cpu | sys       5%  | user     62%  | irq       0%  | idle     31% |  cpu000 w  1% |  guest     0% |  curf 1.20GHz |  curscal  44% |
cpu | sys   7%  | user     58%  | irq   0%  | idle     34% |  cpu002 w  1% |  guest     0% |  curf 1.20GHz |  curscal  44% |
CPL | avg1    3.83  | avg5    4.59  | avg15   4.76  |              |  csw    54101 |  intr   45315 |               |  numcpu     4 |
MEM | tot     7.6G  | free  194.3M  | cache 495.5M  | dirty   2.7M |  buff   38.9M |  slab   86.7M |               |               |
SWP | tot     5.7G  | free    5.5G  |               |              |               |               |  vmcom  12.4G |  vmlim   9.5G |
LVM | nchy-lv_home  | busy     11%  | read       1  | write    524 |  KiB/w      3 |  MBr/s   0.00 |  MBw/s   0.20 |  avio 2.05 ms |
LVM | nchy-lv_root  | busy      1%  | read       0  | write      7 |  KiB/w  4 |  MBr/s   0.00 |  MBw/s   0.00 |  avio 9.00 ms |
DSK |          sda  | busy     11%  | read       1  | write    109 |  KiB/w 19 |  MBr/s   0.00 |  MBw/s   0.20 |  avio 10.3 ms |
NET | transport     | tcpi  72  | tcpo     118  | udpi      15 |  udpo   5 |  tcpao     11 |  tcppo      0 |  tcprs      0 |
NET | network       | ipi       87  | ipo      123  | ipfrw      0 |  deliv     87 |               |  icmpi      0 |  icmpo      0 |
NET | wlan0   ----  | pcki      88  | pcko     123  | si   34 Kbps |  so   19 Kbps |  erri       0 |  erro       0 |  drpo       0 |

  PID  RUID       EUID       THR    SYSCPU    USRCPU   VGROW    RGROW    RDDSK   WRDSK   ST   EXC  S   CPUNR    CPU  CMD         1/4
 3649  saml       saml        34     0.33s    11.98s      0K    1752K       4K   2828K   --     -  R       2   122%  chrome
10399  saml       saml         4     0.14s     3.08s   5120K   -12.8M       0K      0K   --     -  S       2    32%  chrome

Specifically these lines that are showing LVM and DSK activity:

    LVM | nchy-lv_home  | busy     11%  | read       1  | write    524 |  KiB/w      3 |  MBr/s   0.00 |  MBw/s   0.20 |  avio 2.05 ms |
    LVM | nchy-lv_root  | busy      1%  | read       0  | write      7 |  KiB/w  4 |  MBr/s   0.00 |  MBw/s   0.00 |  avio 9.00 ms |
    DSK |          sda  | busy     11%  | read       1  | write    109 |  KiB/w 19 |  MBr/s   0.00 |  MBw/s   0.20 |  avio 10.3 ms |

iostat

If you're interested in the data over a period of time, iostat is still probably your best option, wrapped into a script. You can get rid of the utilization output so that it's easier to deal with:

$ iostat -dx /dev/sda 5
Linux 2.6.35.14-106.fc14.x86_64 (grinchy)   08/23/2013  _x86_64_    (4 CPU)

Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
sda               1.82    96.60    3.84   20.68   122.26   891.99    41.36     0.46   18.77   7.35  18.01

Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00    26.20    1.60   24.60    17.60   350.40    14.05     0.31   11.95   8.84  23.16

Directly from the Kernel

This block of code (bash) will show you the read I/O's processed, directly from the Kernel.

OLD=`awk '{print $1}' /sys/block/sda/stat` # First field is number of read I/Os processed
DT=1
for ii in `seq 1 10`
do
    sleep $DT
    NEW=`awk '{print $1}' /sys/block/sda/stat`
    echo $((($NEW-$OLD)/$DT))
    OLD=$NEW
done

source: https://serverfault.com/a/525248/2518

The information can also be had from either /sys/block/sda/stat or /proc/diskstats in situations where you can't or don't want to install any tools.

References

Thanks for the detailed answer but the metrics you've provided are still I/O focused, ideally, I'd like to see disk seek activity irrespective of I/O throughput — Gearoid Murphy, Aug 23 '13 at 15:45
@GearoidMurphy - I didn't see a way to get this type of telemetry data about disk accesses outside of I/O. — slm, Aug 23 '13 at 15:57
I think Marco hit on it in a comment above, the solution would be to look at the ratio of disk transactions to the actual volume of data read/written. — Gearoid Murphy, Aug 23 '13 at 15:58
@GearoidMurphy - yeah that's why I asked for clarification in the comments as to whether you wanted a real-time type of solution or period of time type. If you come up with a workable solution I'd write it up as an answer and accept it. There wasn't a lot when searching for your particular slant on this. — slm, Aug 23 '13 at 16:03

score 4 · Accepted Answer · answered Aug 23 '13 at 16:56

The ratio (rkB/s + wkB/s)/%util of the iostat -x output should give you some insight:

Device:  rrqm/s wrqm/s   r/s   w/s  rkB/s  wkB/s avgrq-sz avgqu-sz await r_await w_await  svctm  %util
sda        0.04   3.65  7.16  6.37 150.82 212.38    53.71     0.03  1.99    0.82    3.31   0.76   1.03

I'm not sure how exactly this ratio corresponds to the disk seek. But the idea is that, if the disk is busy and does not have a high throughput it is probably seeking. However, it's not guaranteed. Broken disks sometimes show a high utilisation and have almost no throughput. But it's at least an indicator.

You can also provide a number to iostat (e.g. iostat -x 5) to specify the update interval. That way you can monitor continuously.

Presumably, many read requests paired against a low rkB/s would be indicative of such a situation — Gearoid Murphy, Aug 23 '13 at 17:33

How to detect if a disk is seeking?

2 Answers2

nmon

iostat

Directly from the Kernel

References