When a command or process seems to hang, freeze, crash, or whatever; is there a way to investigate exactly what it's currently doing (if anything), behind the scenes?
Would be especially useful to determine if it's crashed, or if it's just carrying out a lengthy task. And if it's not behaving as expected, it could help troubleshoot. Thanks.
There are 2 low level tools you can use to do this, strace, and gdb. The availability of strace is assuming you're on linux or OS where it works. For other OSs, you might have truss, or dtrace, but the technique is similar.
So, strace is easier to use than gdb, and usually answers the question of "what is this application doing" just fine. Typical usage (for me) would be something like:
strace -f -tt -s 200 -p $PID
The only real option you need is the -p $PID. The other options just add more info to the output line, but aren't really necessary.
What this does is show you every system call the application is making. The only way an application can be doing something, and strace not to show it, is if it's purely computational. Meaning crunching numbers or something. Things like reading/writing a file, sending a packet, getting the current time, etc, all require a system call and will show up.
gdb is lower level than strace. gdb will show you what line of code the application is actually running. However there are a few gotchas to this. The big one is that you need the debugging symbols for the application. On distro-provided packages, these have usually been stripped out. However on distros like RedHat derivatives, they are often provided in the form of pkgfoo-debuginfo-1.2.3.rpm. You can just install this package and get the symbols back.
Another thing is that where strace will show you what the application is doing over time, gdb shows you what the application is doing at that exact moment, as it freezes the process while you are inspecting it.
So anyway, usage looks like:
gdb -p $PID
...which will drop you to an interactive shell where you can run something like:
where
or
info threads
I'm not going to go into the details of how to use gdb, as depending on the complexity of your application, it could be complex, and there are plenty of guides on the internet.
When you're done poking at the process, and want to let it resume, just use quit or CTRL+D.
You can start most processes from a terminal window with verbose output by appending a -v flag. That way you will see what (if anything) is happing in the terminal window when it appears to hang.
top and htop provide useful information about running processes, including updated info about mem and cpu usage, as well as some very useful summary info also.
You might also take a look at the ps command docs with man ps.
