23

Why does apt-get not use 100% of either cpu, disk, or network -- or even close to it? Even on a slow system (Raspberry Pi 2+) I'm getting at most 30% CPU load. I'm just thinking that either it's being artificially throttled, or it should max out something while it's working ... or it should be able to do its thing faster than it does.

Edit: I'm just measuring roughly via cpu/disk/net monitors in my panel, and the System Monitor app of Ubuntu MATE.

Please explain why I'm wrong. :-)

Update: I understand that apt-get needs to fetch its updates (and may be limited by upstream/provider bandwidth). But once it's "unpacking" and so on, the CPU usage should at least go up (if not max out). On my fairly decent home workstation, which uses an SSD for its main drive, and a ramdisk for /tmp, this is not the case.

Or maybe I need to take a closer look.

KlaymenDK
  • 649
  • 1
  • 7
  • 21
  • How are you measuring disk and network load? – JigglyNaga May 25 '16 at 11:20
  • 1
    Disk IO is just like network IO, though. It will still block the app, preventing it from using the CPU. Alas, `apt-get` isn't particularly good at optimizing this. I imagine it could install as it downloads so that by the time your download is finished most of your payload could already be installed, but, unfortunately, it doesn't. In any case, standalone installs mostly just extract data to disk. Those operations are inherently IO bound, and there's simply not much else to do but wait on the disk drive to finish reading or writing. – Petr Skocik May 25 '16 at 12:02
  • How did you get the *30% CPU load* number? – A.L May 25 '16 at 14:23
  • 1
    @PSkocik "I imagine it could install as it downloads" apt-get just downloads, dpkg installs. And dpkg is smarter than apt-get in the order that a bunch of packages should be installed, which may not be the same that apt-get downloads them. – Braiam May 25 '16 at 17:47
  • Note that an application which is 100% CPU bound for half a tick, and then 100% IO-bound for the other half will appear neither CPU-bound nor IO-bound,. – MSalters May 26 '16 at 11:53
  • @Braiam: A hypothetical `apt-get` that's smart enough to pipeline `dpkg` installs would know which packages to download first (those without additional dependencies). As apt-get understand dependencies, and the dependency graph is acyclic, there is always one package which can be installed first, and this should be downloaded first. – MSalters May 26 '16 at 11:55
  • @MSalters at which point you wonder, why should two simple straightforwards tools over-complicate a process, when even the maintainer has other [more important (and high impact) things to do](http://www.chiark.greenend.org.uk/~cjwatson/blog/no-more-hash-sum-mismatch-errors.html)? – Braiam May 26 '16 at 11:58
  • @Braiam: At my previous company we reused `dpkg` and had to rewrite it, for use on embedded systems where we couldn't afford the disk overhead of downloading every package before installing it. This meant we needed 3x the disk space: old install, new packages, new install. Our integrated tool could stream the packages, install them on the fly, and was smart enough to handle the few exceptions (kernel updates & the tool itself, mainly).A real problem with the Linux philosophy of "2 tasks, 2 programs" is avoidable communication overhead. – MSalters May 26 '16 at 12:21
  • You could try apt-fast. It might speed up the download portion of the task, but it probably wouldn't affect the rest. – Joe May 28 '16 at 00:57
  • Everything using the CPU always uses 100% of the CPU, for a short period of time. If that "period" is a third of a second, then we say it used the CPU 33% of the time. So if a process spends some time on the CPU, some time waiting on net and some waiting on disk, that will basically make up (approximately - there are other resources and processes can wait for other processes or for a resource to become available), the whole. – Johan Jun 01 '16 at 12:35

4 Answers4

29

Apps will only max out the CPU if the app is CPU-bound. An app is CPU-bound if it can quickly get all of its data and what it waits on is the processor to process the data.

apt-get, on the other hand, is IO-bound. That means it can process its data rather quickly, but loading the data (from disk or from the network) takes time, during which the processor can do either other stuff or sit idle if no other processes need it.

Typically, all IO requests (disk, network) are slow, and whenever an application thread makes one, the kernel will remove it from the processor until the data gets loaded into the kernel (=these IO requests are called blocking requests).

Petr Skocik
  • 28,176
  • 14
  • 81
  • 141
  • 6
    With `apt` commands, it's aggravated by the fact that many files are open in sync mode, or with frequent explicit flushes to disk being requested to guarantee data on disk stays in a consistent state as a system crash could have serious consequences otherwise. Running `apt` commands with `eatmydata` can often dramatically improve performance at the expense of reduced reliability (not to mention that services started as part of package installations will inherit the eatmydata settings) – Stéphane Chazelas May 25 '16 at 14:04
  • Lol at that last point :). Does anyone have numbers for eatmydata since the 2010 commit in https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=578635 ? I don't know if "dramatically" is the right word still. – sourcejedi May 25 '16 at 15:02
  • Ah, maybe it is (at least on some cloud providers) https://bugs.launchpad.net/cloud-init/+bug/1236531/comments/6 – sourcejedi May 25 '16 at 15:08
  • 1
    @sourcejedi On a Raspberry Pi2 with a relatively high-end SD card (but still an SD card, not a high-end SSD), I consider “dramatically” to be a bit of an understatement. The performance of dpkg on flash media really sucks. – Gilles 'SO- stop being evil' May 25 '16 at 22:27
  • 1
    If it's disk-IO-bound then why is it not using 100% disk bandwidth? – user253751 May 26 '16 at 01:40
  • @immibis IO bound doesn't necessarily mean that the process wants to do huge amounts of IO, it simply says that it's not really doing many computations and *most* of its work is reading/writing to disk. In any case if you want an answer is that the OS has schedulers and buffers precisely to avoid one process taking over all the system. If you want a system where you can define a process as X-bound and be sure it will use 100% during operation you have to move to systems without multitasking. – Bakuriu May 26 '16 at 14:08
  • Because you can't really define 100% disk bandwidth. Even solid state drives have varying performance across various file or stripe sizes. – Colton McCormack May 26 '16 at 14:41
  • When I use apt on a system running in RAM, I surely reach 100% CPU with apt-get. It can't take advantage of multiple CPUs though (no multi-threading). – pedroapero Oct 11 '18 at 11:56
16

Even on a slow system (Raspberry Pi 2+) I'm getting at most 30% CPU load.

The Raspberry Pi 2+ has 4 cores. For some monitoring tools, a 100% usage correspond to all the cores been used at 100%. If only one core in a quad code processor is used, the CPU load is 25 %. The 30% CPU load you mention is roughly one core used at 100% while some processes are running on the other cores:

(100% on one core out of 4 = 100 / 4 = 25%) + some processes ≃ 30%

Since apt-get is not multi threaded, it will never use more than one processor, which is 25% of all the CPU resources.

Here is an example on my 8 cores (4 cores with Hyper-Threading) machine running Ubuntu, I launched one thread with the cat /dev/zero > /dev/null command in order to create an infinite process that utilize one core entirely.

Now if we take a look at the graph from htop, we can see that the average load (Avg bar) is 12.7%, which correspond to one core used at 100%, which is also 1/8 of all the CPU resources:

(100% = 100 / 8 = 12.5%) + some background processes ≃ 12.7%.

htop

It can also be noted that the command has a value of 100% in the CPU% column, this is because it's relative to one core and not to all the cores.

A.L
  • 1,492
  • 3
  • 15
  • 29
  • +1, a usage % close to a multiple of (100 / nCores) should always trigger further scrutiny. This can be checked - and indeed is precluded - by using a monitor able to show usage per core, where 0 <= the% <= 100 * nCores – underscore_d May 25 '16 at 21:39
  • Isn't `/dev/zero > /dev/null` a better example, since urandom will deplete the entropy pool? – Filip Haglund May 26 '16 at 13:09
  • @FilipHaglund `cat /dev/zero > /dev/null` gives the same result, I didn't know that device, thanks. *urandom will deplete the entropy pool* I don't know the entropy pool, how can it be a problem? – A.L May 26 '16 at 13:49
  • 1
    When programs use crypto, they need truly random data to generate secure encryption keys. The computer generates entropy by watching the mouse move among other things. There are hardware random number generators, but most computers don't have them. If the entropy is all used up, the code that needs secure entropy has to wait for more to be generated. Urandom will use truly random bits if available, or otherwise return less secure random bits. – Filip Haglund May 26 '16 at 13:56
  • *When programs use crypto* Even if I think that nobody will perform a CPU benchmark while generating a random key, I've updated my answer as a precaution. – A.L May 26 '16 at 14:08
2

I think you're actually not measuring IO %. I haven't seen a Linux IO% widget. (I'm very envious of the Windows 10 task manager :). Check using the iotop command and you will see 100% IO.

top should show 100% across user+system+iowait, for values of 100% divided by your core count as described by A.L. I'm not saying top is 100% helpful, but it can be a really useful all-around tool to learn.

Throughput will be lower than maximum, because you're unpacking lots of small files, aka "random IO". There's also some disk sync / cache flushes, although since 2010 on Linux there's only a few of them for each package installed. (Used to be one per file).

sourcejedi
  • 48,311
  • 17
  • 143
  • 296
  • Use `iotop --only`, the `--only` option *only show processes or threads actually doing I/O*. – A.L May 25 '16 at 13:36
  • 4
    iostat, dstat, atop... will show per disk disk utilisation without needing privileges. It's for the per-task utilisation that you need privileges – Stéphane Chazelas May 25 '16 at 14:12
  • @StéphaneChazelas absolutely correct. The point I was trying to make (ninja edit) is that the OP mentions a couple of GUI tools. And the particular GUI tools I've seen, like Gnome System Monitor, show throughput but no IO%. – sourcejedi May 25 '16 at 14:56
2

Actually, IO/Network requests are really slow compared to CPU ops. This means that while your network card is fetching data, or your disk is writing this data, your CPU does absolutly nothing (for this process anyway).

If your hard drive is speeder than your network connection (which is probably true), it won't write more than it has received.

Finally, the network percentage corresponds to the max possible network card usage, not connection. So you may have a 1Gb/s network adapter, you're really unlikely to have an internet connection that reaches this bandwidth.

Aymeric R.
  • 121
  • 2