how to find out the full path of the command from the result of lsof -i

Question

lsof is a great utility, just now started using it.

lsof -i | grep smtp => this give the following result.

httpd.pl  212548          global    3u  IPv4 893092369      0t0  TCP server07.host...blah...

In the above example, httpd.pl is perl script, which sends spam emails.

How can I know the full path of the command. i.e in the above result, I want to know the full path of httpd.pl

I tried searching in home dir, the file httpd.pl is not there. and also, I tried with lsof -p PID, this also does not give the path to it.

Is there any way, I can get the full path of that file ?

Note: This kind of problem is very common in shared hosting environment. So, it will be very useful for shared hosting or any web server administrator.

Answer 1

One way could be to examine the files opened by that process. Of the types shown in the FD column of lsof:

FD         is the File Descriptor number of the file or:

               cwd  current working directory;
               ...
               txt  program text (code and data);

So, try:

lsof -a -d txt -p 212548

For scripts, this does show the path of the interpreter used (such as /bin/bash). For shell scripts, the script file seemed to be open on fd 255 on my system, but for Perl scripts, there was no mention of the script file at all in lsof output.

Answer 2

That httpd.pl there is the name of the process. That's initialised from the base name of the file that the process last executed, but can also be modified by other means.

With perl, it's just a matter of assigning a value to $0:

$ perl -e '$0 = "httpd.pl"; sleep 10' & ps -fp $!
[2] 11954
UID        PID  PPID  C STIME TTY          TIME CMD
stephane 11954 31685  0 09:57 pts/8    00:00:00 httpd.pl

So, there's no telling whether that httpd.pl comes a file called like that or something else. perl could still have been run with code passed as -e or from another file or from stdin...

If the perl code was loaded from a file called httpd.pl (or anything other file for that matters), you could probably not tell either because perl usually closes the file after it has read its content, so it wouldn't show up in the output of lsof -p PID.

If the script was indeed passed to the perl interpreter and if the perl code doesn't subsequently modify the process name or the arguments, you may get a clue by looking at its command line arguments with ps -fp PID.

You would then see the path of the file as passed as argument. It may be a relative path, in which case you can look at the cwd entry in lsof which would help unless the script changes the current working directory.

If the script was started by a shell, you may also be able to find the path (again possibly relative) in the _ environment variable (grep -z '^_=' /proc/PID/environ on Linux).

In any case, the script could have very well deleted itself first thing upon starting.

If you want to see the code of that httpd.pl, best would be to dump the content of that process' memory (like with gcore, shipped with gdb or looking at /proc/PID/{maps,mem} on Linux). And look for the code in there.

For example, looking for #! in NUL-delimited records:

perl -e '$p=shift;open MAPS, "/proc/$p/maps";
  open MEM, "/proc/$p/mem" or die "open mem: $!";
  for $m (grep !m{/|\[v}, <MAPS>){
    ($a,$b) = map hex, $m =~ /[\da-f]+/g;
    seek MEM, $a, 0 or die "seek: $!";
    read MEM, $c, $b - $a or die "read: $!";
    print $c
  }' PID | grep -z '#!'

YYMV though as the raw source code may no longer be in there. You'd then need to look for the pre-compiled code and decode it.

This stackoverflow Q&A would then help you doing that.

Answer 3

First of all, you don't need pipe through grep: lsof -i:$PORT selects the listing of files any of whose Internet address uses the given $PORT number (or service name), so

lsof -i:smtp

will suffice in your example.

If instead you already know the command name, lsof -c $COMMAND selects the listing of files for processes executing the command named $COMMAND, for example:

lsof -c httpd.pl

Third notion: f you know the PID of the process, you can get its full command path with

ps -p $PID -o command

where $PID is its PID number.

So if you need to find out the full path of the command from the result of lsof you could either use:

ps -p $(lsof -i:$PORT |awk 'NR>1 {print $2}' |sort -n |uniq) -o comm

or

ps -p $(lsof -c $COMMAND |awk 'NR>1 {print $2}' |sort -n |uniq) -o comm

Notice 1: awk, sort, uniq are necessary to skip the header line from the output and to aggregate the listing by PID number.

Notice 2: If you use a destination port $PORT that is in use by multiple processes (for example https), you will need to parse each line separately:

for PID in $(lsof -i:$PORT |awk 'NR>1 {print $2}' |sort -n |uniq); do
  ps -p $PID -o comm
done

Answer 4

By itself, lsof -i will only list Internet information. You can add file information (or show that instead) using the -d option. For example:

lsof -d txt | grep -E '/httpd.pl$'

because (unless a process changes the name under which it runs) the command name will match the actual filename which was loaded.

That file has to be executable, of course — but so could the shared libraries it loads. In a really pathological case, you could have a program running from a script (written in Python for instance) which loads shared libraries — and renames itself in the process list.

According to a CentOS thread Suspicious Process httpd.pl, there was a process started from a PHP script which called itself httpd.pl:

For anyone that was interested. This was a backdoor installed by a php shell. the cleanup just didn't catch it on the first round.

Another thread, Sucpicious process running in CentOS server gives more details:

The httpd.pl is a spoofed process that is running perl.

it is usually a backdoor that is set up by a php shell that will be loaded into a website somewhere

Further reading:

• How can a process appear to have different name in ps output?
• change process name without change argv[0] in Linux