9

A few of my processes consume 100% cpu. I'm trying to figure out which scripts are causing it

I tried running strace ps -ef:

open("/proc/PID/status", O_RDONLY) = 6
read(6, "Name:\textract\nState:\tR (running)"..., 1023) = 1023
close(6) = 0
open("/proc/PID/cmdline", O_RDONLY) = 6
read(6,

So it gets stuck trying to read /proc/PID/cmdline. I tried catting that, and it got stuck again. Something is obviously screwed in the kernel; what should I try next?

Note: rebooting doesn't work -- if I shutdown manually the problem starts again. I'm using SUSE Linux Enterprise Server 11 (x86_64), Linux 2.6.27.19

Edit: ps -e produces output, and I found there are too many greps. The number of greps varies: 250, 450, and now I see around 520 greps. I traced back and found it is the result of a cron script. I still have to understand those cron scripts. Yes, top displays results. We manually shutdown the server 2 days back. System has been running from last 2 days. I see some oracle stuff running all the time. I just did the memory test, no faults detected

Stéphane Chazelas
  • 522,931
  • 91
  • 1,010
  • 1,501
72616b657368
  • 121
  • 1
  • 4
  • What distribution and version? What kernel version? What architecture? Does `ps -e` produce output? What about `top`? Has this system been running for a long time, what is it normally running? Have you done a memory test recently? Not being able to read `/proc/$pid/cmdline` looks like a kernel bug or an exploit. – Gilles 'SO- stop being evil' Apr 09 '11 at 21:52
  • What are the processes that consume most CPU? The bunch of greps? Perhaps the hog is some badly written query to Oracle? – vonbrand Jan 15 '13 at 16:01

2 Answers2

2

Had that just yesterday. The problem was, one process was in "uninterruptible sleep" state, shown as status D in top. ls /proc/ does not return and is not abortable. ps -ef does not return and is not abortable.

If rebooting does not help you probably have a bad sector on your DVD or hard disk and the process PID is trying to read there during startup. So technically rebooting helps, but the error re-occurs automatically.

Check with top if the process is indeed in status D, then go on from there. Boot the computer without calling this process (rescue system). Then start the program stracing it and see which files it accesses. I bet one file has bad sectors.

Thorsten Staerk
  • 3,687
  • 1
  • 22
  • 28
  • 1
    This is exactly what was happening to us at work today. We noticed a process stuck in uninterruptible sleep *D*, adjusted logging to see more details, reran, and then ps -ef and pgrep both started hanging. We noticed it inside podman container, but eventually noticed, even the host OS could no longer `ps -ef` without hanging. – Ajax May 25 '23 at 01:32
0

It looks that grep is hanged and due to cron job scheduling another process will become active after certain amount of time (as written in crontab). Multiple processes will lead an unresponsive system

Try following debugging method :

  • Change crontab entry for increasing script interval (so that your hanging script does not get executed many times)
  • Record output of top for an interval
  • Traverse the process tree from top logs and then find the process at which it is hanging
  • Then traverse the code form where the same thing is being called.
shubham
  • 407
  • 3
  • 12