22
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          11.50    0.02    5.38    0.07    0.00   83.04

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
sdc               0.01    89.92    0.26   41.59     3.36   457.19    22.01     0.23    5.60   0.09   0.38
sdb               0.10    15.59    0.40   14.55     8.96   120.57    17.33     0.04    2.91   0.07   0.11
sda               0.13    45.37    0.96    8.09    20.06   213.56    51.63     0.02    2.64   0.16   0.14
sde               0.01    31.83    0.09   11.34     0.94   103.56    18.29     0.04    3.52   0.14   0.16
sdd               0.01    48.01    0.13   19.81     1.58   202.16    20.44     0.11    5.62   0.13   0.25

Is there a way to know what files are being written? 457 kB/s

Also this other linux machine have this same problem.

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          20.50    0.00   46.48   20.74    0.00   12.28

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
sda               0.17    11.61    0.99    3.51    36.65    59.43    42.70     0.10   23.20   3.84   1.73
sdb               0.55   224.18   24.30   97.45   246.48  1287.12    25.19     3.96   32.53   7.88  95.91
sdd               0.53   226.75   25.56   90.96   283.50  1271.69    26.69     3.43   29.44   8.22  95.75
sdc               0.00     1.76    0.28    0.06     4.73     7.26    70.41     0.00   12.00   2.12   0.07
dm-0              0.00     0.00    1.11   14.77    36.41    58.92    12.01     1.00   62.86   1.09   1.74
dm-1              0.00     0.00    0.04    0.12     0.17     0.49     8.00     0.00   21.79   2.47   0.04
dm-2              0.00     0.00    0.01    0.00     0.05     0.01     8.50     0.00    7.90   2.08   0.00

1200 write request per second for a server that host nothing

user4951
  • 10,329
  • 28
  • 71
  • 92

5 Answers5

13

Well, you could try the following commands which worked for me in RHEL6:

  1. Whatever device you see in "iostat" output performing more I/O, use it with fuser command (from the psmisc package) as follows:

    fuser -uvm device

  2. You will get a list of processes with the user name causing more I/O. Select those PIDS and use it in the lsof command as follows:

    lsof -p PID | more

  3. You will get a list of files/directories along with the user performing maximum I/O.

Stephen Kitt
  • 411,918
  • 54
  • 1,065
  • 1,164
Ravi Trivedi
  • 298
  • 2
  • 7
  • root@host [~]# fuser -uv /home4 root@host [~]# fuser -uvm /home4 USER PID ACCESS COMMAND /home4: newgames 18297 ..c.. (newgames)php cuntschi 18407 ..c.. (cuntschi)php newgames 18457 ..c.. (newgames)php – user4951 Jan 06 '13 at 08:29
7

It seems there is no tools to find out I/O throughput per file other than within process using the file. But there are ways to find out process I/O throughput.

iotop - It is a top/iftop like utility that show process I/O throughput.

After pin pointing which process is having heavy I/O, use following to find out what file is being used

lsof -c <process name>

That should narrow down the scope and help pin point the source.

John Siu
  • 4,695
  • 2
  • 25
  • 22
  • how to get the process? lsof also produce nothing. Something is writing so many things and I don't know what – user4951 Jan 06 '13 at 04:48
  • Sorry, I put the wrong io utility. It should be `iotop`, not `iostat`. I corrected my answer. – John Siu Jan 06 '13 at 04:50
  • (1) Can you also update what distro(redhat/centos/ubuntu ...) you are running? (2) Is this a fresh install? (3) Any file sharing like samba running? – John Siu Jan 06 '13 at 04:57
4

You can use inotifywait from inotify-tools to find out exactly which file is being written to. This won't tell you how much data is being written, but it will at least tell you what files are being written to as it happens.

As an example, this command will print the file name as soon as any are created, modified, or deleted in /tmp:

$ sudo inotifywait -e modify -e attrib -e move -e create -e delete -m -r /tmp
Setting up watches.  Beware: since -r was given, this may take a while!
Watches established.
/tmp/ CREATE test
/tmp/ MODIFY test

Unfortunately you will need to guess which directory contains the files being written to. This fails if you try to use it on the root directory, though apparently that can be overridden:

$ sudo inotifywait -e modify -e attrib -e move -e create -e delete -m -r /
Setting up watches.  Beware: since -r was given, this may take a while!
Failed to watch /; upper limit on inotify watches reached!
Please increase the amount of inotify watches allowed per user via `/proc/sys/fs/inotify/max_user_watches'.
adittes
  • 41
  • 1
2

Use systemtap to monitor each write/pwrite syscall. You can do some accounting and you will see which file/fd gets the most bytes.

Erben Mo
  • 121
  • 2
2

csysdig. https://github.com/draios/sysdig/wiki/Csysdig-Overview

They even have a link to a video demo-ing how to do it. https://www.youtube.com/watch?v=UJ4wVrbP-Q8

bbrendon
  • 121
  • 4