5

I have a workstation(2x Intel Xeon family CPUs and 128GiB of RAM) running several virtual machines and while the combined CPU usage is <30%, then the load average is between 20 and 25. For example, if I execute a tar -xzvf vm_data.tgz --directory vm4/ --strip-components=1 command, then the gzip process is 90% - 99% of its time blocked by I/O and the command takes forever to complete:

enter image description here

On the other hand, the actual reads and writes to disks are very low compared to SATA 3.0 or SSDs(I'm using single Kingston SA400S37960G SSD) hardware limits.

What might cause a process(gzip in my example) to wait after the I/O while the actual disk reads and writes appear to be very low? My first thought was that maybe the system interrupts are very high and that's what's blocking the I/O, but according to /proc/interrupts this does not seem to be the case as none of the counters are increasing rapidly.

Martin
  • 7,284
  • 40
  • 125
  • 208
  • 2
    What filesystems are involved? – roaima Sep 21 '20 at 08:26
  • 1
    I have a similar problem, and I don't know the reason, but dropping caches with `echo 3 | sudo tee /proc/sys/vm/drop_caches` or similar unblocks it. This may or may not work for you. – dirkt Sep 21 '20 at 09:44
  • 1
    @roaima `ext4` and I'm operating within the same partition/file system. @dirkt unfortunately, dropping the cache does not help. – Martin Sep 21 '20 at 13:03
  • 1
    @Martin, Do the `qemu` instances use large files, which are virtual hard disks with partitions and everything, or large files which are just file systems, or, something else? (IIRC `qemu` has at least one or two special formats, for saving space, or acting something like a live `diff` or *delta* file-system.) Also is that Kingston SA400S37960G the *only* storage device on the system? – agc Sep 21 '20 at 13:22
  • 1
    To simplify the previous questions: please provide the command line invocations of the `qemu` instances. – agc Sep 21 '20 at 13:37
  • 1
    what is the size of tgz file? – Sandeep Kothari Sep 22 '20 at 14:34
  • @agc Yes, they use either 1.3GB `qcow2` disks which hold four partitions or 2.2GB `raw` disks which hold six partitions. The Kingston SA400S37960G is the only storage device in the system. @Sandeep Kothari Size of the tgz file is 4.6GB. – Martin Sep 27 '20 at 19:24
  • Here's a new idea. What kind of disks are you dealing with, are those SSDs or HDDs? In case of HDDs if any of your VMs does a lot of random IO operations, this could slow down everything else to a crawl. Also, while looking at your screenshot I see that gzip's write speed is around 3.6MB/sec. Considering your overall CPU usage I cannot say that's too low, that's probably in line with your CPU performance. Can you temporarily suspend the execution of your VMs and check if gzip runs significantly faster? – Artem S. Tashkinov Sep 28 '20 at 00:51

3 Answers3

4

Starting from your report displaying ~40G free memory (I know that it is not exactly the amount of available memory but, let's hold for quick & dirty calculus 40G available. And 12G taken by buffer/cache, that, because of no detailed info provided we will admit full of... dirty data.

Because vm.dirty_ratio defaults to 20% and 20% of 40G = 8G < 12G...

I suspect that your system is running over the limit commanding the process to writeback by itself. In other words issuing blocking writes.

I'd then check what are actually the system's limits :

$ sysctl -a | grep dirty

And if you discover that vm.dirty_ratio is actually to it's default value, increase it. (You can easily go up to 80% without worrying, If I recall correctly Oracle has been recommending this value.)

While you are at it, you can also lower it's companion (vm.dirty_background_ratio) which usually default to 10. Low latency system recommends the lowest possible value (1), I personally set this one to 3. This will enable the writeback daemon to operate sooner delaying the point when the cache will pass over the limit fixed by the dirty_ratio.

You can make temporary changes by echoing values into corresponding entry of the /proc/sys/vm/ directory structure. In order to make these changes permanent (across reboots) you can edit /etc/sysctl.conf

This being the immediate cause for which the process blocks, then for the reason why writing to the device seems so slow to the point of filling the cache above the dirty_ratio limit : see artem-s-tashkinov answer.

MC68020
  • 6,281
  • 2
  • 13
  • 44
  • I set the `sudo sysctl vm.dirty_background_ratio=3` and `sudo sysctl vm.dirty_ratio=80`, but I the `tar` command in my initial post did not run faster. – Martin Sep 27 '20 at 19:44
  • A/ Which i/o scheduler are you running ? B/ Could you post the ouput of the cat of /sys/block/*-the-physical-device-involved-*/nr-requests , read_ahead_kb, rotational ? – MC68020 Sep 28 '20 at 11:18
3

I had a very similar issue many years ago with our production MySQL database. It turned out its files very extremely fragmented and backing them up resulting in all other disk operations taking forever to complete.

Please post the output of:

find vm4 -type f | while read filename; do sudo filefrag "$filename" | egrep -v ": 1 extent|: 0 extents"; done | sort

To resolve the issue, in case my guess turns out to be true, you'll need to defrag VM files.

Artem S. Tashkinov
  • 26,392
  • 4
  • 33
  • 64
  • Requests for further clarification should be done in comments under the Question, not as part of an Answer. Save Answers for when you know the problem and are addressing it. If you feel like file fragmentation is the answer, just leave that portion of it. Thank you! – Jeff Schaller Sep 24 '20 at 13:49
  • I will delete the answer in case it's not applicable to the OP issue. Posting this much info as comments won't look pretty. Also, it would be easier to get replies from the OP here under this answer instead of having everything lumped together in the comments. Also you cannot have new lines and coloring in the comments. – Artem S. Tashkinov Sep 24 '20 at 13:53
  • 1
    Perhaps you could explain what you're looking for with the output -- how to interpret whether the output means your answer would apply or not? – Jeff Schaller Sep 24 '20 at 14:04
  • `filefrag` will report how many fragments the files occupy. If we are talking about too many fragments (the maximum would be in Python `filesize//4096 - 1`), it could explain the system sluggishness. – Artem S. Tashkinov Sep 24 '20 at 14:06
  • @ArtemS.Tashkinov Number of extents for all the disks used by virtual machines is between 2 and 37. – Martin Sep 27 '20 at 19:41
  • @Martin That's great which unfortunately renders my suspicion invalid and leaves me puzzled. – Artem S. Tashkinov Sep 28 '20 at 00:47
2

I noticed this after switching from Centos7 (3.10.0.1062) to ElRepo kernel-ml (5.11.6) : There was an approximately 10x improvement in writeback speed (from 2 Mb/sec to 20Mb/sec) but I was using a Samsung NVMe 400G drive capable of over 2000 Mb/sec, so I expected better. I then switched the ext4 mount to "nodelalloc,dioread_nolock" and now I get writeback speeds of over 1500Mb/sec. Since I have 512GB memory and have dirty_background_ratio = 10, dirty_ratio = 20, while untaring a large folder with many medium (30-50kb) files, the dirty pages grow to over 50 GB (at a rate of around 500Mb/sec, limited by my source HDD RAID speed), then all of the sudden writeback happens at 1500+Mb/sec for around 35+ seconds. During the 35+ second writeback period, the tar job drops to 0% CPU and 0% IO. Something about the writeback kernel thread must require a lock that is also required by the tar write system call, and this behavior was definitely not present in Centos7 (3.10.0.1062).

So the behavior described by OP is real, but seems to have been introduced recently.

Thomas Anantharaman
  • 21
  • 1