3

I went thru a problem where I could only guess what process was eating the cpu.

My cpu usage was about 80% for all cores at psensor.

I tried htop, top and ps -A -o pcpu,pid,cmd --sort +pcpu (the last one I even tried with sudo to no avail).
All these shown the culprit pid (that I was aware of) using about 7% only...

When I SIGKILL on that pid, all get back to normal.

To test, I did an infinite loop on terminal while true;do echo -n;done but that I could clearly see at htop; so my guess what was causing trouble was not similar to that...

So I wonder if there are other ways I could have found the culprit without having to guess?

Thinking again, I think I would like to know what calculations psensor and "system load indicator applet" uses that was able to show that value but the others were unable to?

PS.: linking about wait time, linking about load average

Community
  • 1
Aquarius Power
  • 4,099
  • 5
  • 38
  • 56
  • Which process was eating CPU time? Do you have an idea of what it was doing? What did psensor and the system load indicator applet report about this process? How did you find out the culprit? – Gilles 'SO- stop being evil' Jun 22 '14 at 22:06
  • @Gilles I was coding nautilus scripts but nautilus was freezing so I had to SIGKILL it; nautilus had spawned a child running my script but it hadnt kicked in (it spawns a xterm but nothing showed up, so my guess is it broke before my script being run (may even be on its very beginning) (anyway it is fully working now and I still cant reproduce the problem)); I had 4 processes making a cpu load avg of above 20 (I think it was 40)! I SIGKILL 3 of them and missed one; so I came to this question; after I looked more then I found the process, SIGKILL it and all went back to normal. – Aquarius Power Jun 22 '14 at 23:03

3 Answers3

4

I am not familiar enough with the details to give precise hints but I guess there are two sources of differences between the real caused load and the shown CPU usage:

  1. The process may consist of several threads and top may not sum them up. You can see the number of threads by this:

    ps -eo pid,nlwp,%cpu,user,args

    In top you can switch the thread handling with H. The CPU usage of each thread is usually quite low.

  2. The process may cause a lot of I/O. I/O wait time is part of the overall CPU load but may not be part of a process's CPU usage value. So check the wait value in top. It does not tell you which processes cause it to which extent but if the value is low then it cannot explain the effect.

Hauke Laging
  • 88,146
  • 18
  • 125
  • 174
  • `htop` shows threads by default! :), btw I would like to link [this question](http://serverfault.com/questions/155882/wa-waiting-for-i-o-from-top-command-is-big) about tracking wait value – Aquarius Power Jun 22 '14 at 19:23
  • one thing I must say, despite showing high cpu usage, the system was pretty usable, almost not lagging (not sure that is a clue anyway..) – Aquarius Power Jun 22 '14 at 19:38
  • I think, `psensor` uses another way to calculate cpu usage btw – Aquarius Power Jun 22 '14 at 19:41
  • @AquariusPower Maybe the CPU usage is high but at low priority so that the system does not seem slow when you use it interactively. – Hauke Laging Jun 22 '14 at 20:01
  • the process I SIGKILL was normal priority, anyway I tried this `nice -n 19 bash -c "while true;do echo -n;done"` and it was clearly seen on htop as using 100% (may be of one cpu only as the overall cpus usage wasnt above 40% on `psensor`); this seems more tricky now, mostly considering I am being unable to recreate the situation.. I will take a look on psensor source code see if I can make something out of that.. – Aquarius Power Jun 22 '14 at 22:37
3

The code executed on a unix system is classified as either kernel code or user land code. User land code is always attached to a process, so if the CPU is busy executing user land code, it shows on some line in top. Kernel code is normally attached to a process: if the kernel is executing a system call, then the in-kernel processing is accounted as belonging to that process. Kernel time is the “system time” reported by the time utility.

Some of the things that the kernel does can't be directly accounted against one process. In particular, hardware interrupts don't intrinsically belong to a particular process. For example, suppose that an interrupt is triggered by the network card. The kernel executes code to read and parse the network packet; so far no process is involved. The packet may be rejected via a firewall rule, in which case no process can claim that processing time. If a process ends up receiving that packet, some of the reception time will be put on that process's tab, but not the early stages.

So it's possible to have CPU time that doesn't belong to any process. Yet sometimes that CPU time is indirectly caused by some process. For example, if there's a process that sends packets to another machine and causes this other machine to reply, but the firewall blocks the reply packets, then the time spent parsing and discarding the reply packets won't be traced back to that sending process; but if the sending process stops, which causes the remote machine to stop replying, then the kernel won't spend time rejecting the packets any more. Of course the network is just one example, there are many other ways for the kernel to do things that can't be traced directly to one process.

You haven't given enough information to be sure that this is what is going on (and it can be difficult to figure out without a kernel debugger), but this is a plausible explanation.

Gilles 'SO- stop being evil'
  • 807,993
  • 194
  • 1,674
  • 2,175
  • I will try to install a kernel debugger to try to track kernel cpu usage independently of processes; I tried the systemtap (stap) and it crashed with "missing x86_64 kernel/module debuginfo", I think it may be installed and both (kernel debugger and stap) will work. And indeed, if I could at least reproduce the problem it would be easier to provide more further accurate info.. nautilus is not failing a single time on my scripts now... – Aquarius Power Jun 23 '14 at 01:13
  • I'd like link [this](http://askubuntu.com/a/173295/46437) – Aquarius Power Jun 23 '14 at 01:17
2

If don't want to use htop,ps,top, you can use systemtap, for more low level details

c4f4t0r
  • 649
  • 5
  • 11
  • any idea what script I could use with `stap`? I tried [iotop.stp](https://sourceware.org/systemtap/examples/io/iotop.stp) that crashed, I think a reboot may fix the missing kernel module tho; I also found these interesting: disktop.stp, nettop.stp, topsys.stp – Aquarius Power Jun 23 '14 at 00:43
  • One might start with something like syscall-level tracing, e.g. the process/forktracker.stp or procmod_watcher.stp samples, to see if there are many short-lived processes (which may be invisible to sampling-based tools like top). Maybe try broader syscall stats like thread-business.stp. Maybe try systemwide profiling like profiling/pf3.stp. See https://sourceware.org/systemtap/examples/keyword-index.html – fche Jun 26 '14 at 00:01