20

I'm somewhat baffled that inside a Docker container lsof -i doesn't yield any output.

Example (all commands/output from inside the container):

[1] root@ec016481cf5f:/# lsof -i
[1] root@ec016481cf5f:/# netstat -tulpn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      -
tcp6       0      0 :::22                   :::*                    LISTEN      -

Please also note how no PID or program name is shown by netstat. fuser also gives somewhat confusing output and is unable to pinpoint the PIDs as well.

Can anyone shed any light on this?

  • How can I substitute lsof -i (to see the process name as well!)
  • Why is the output of netstat crippled as well?

NB: The container runs with "ExecDriver": "native-0.1", that is Docker's own execution layer, not LXC.

[1] root@ec016481cf5f:/# fuser -a4n tcp 22
Cannot stat file /proc/1/fd/0: Permission denied
Cannot stat file /proc/1/fd/1: Permission denied
Cannot stat file /proc/1/fd/2: Permission denied
Cannot stat file /proc/1/fd/3: Permission denied
Cannot stat file /proc/1/fd/255: Permission denied
Cannot stat file /proc/6377/fd/0: Permission denied
Cannot stat file /proc/6377/fd/1: Permission denied
Cannot stat file /proc/6377/fd/2: Permission denied
Cannot stat file /proc/6377/fd/3: Permission denied
Cannot stat file /proc/6377/fd/4: Permission denied
22/tcp:

(I am not obsessed by the Permission denied, because that figures. What confuses me is the empty list of PIDs after 22/tcp.)

# lsof|awk '$1 ~ /^sshd/ && $3 ~ /root/ {print}'
sshd    6377      root  cwd   unknown                        /proc/6377/cwd (readlink: Permission denied)
sshd    6377      root  rtd   unknown                        /proc/6377/root (readlink: Permission denied)
sshd    6377      root  txt   unknown                        /proc/6377/exe (readlink: Permission denied)
sshd    6377      root    0   unknown                        /proc/6377/fd/0 (readlink: Permission denied)
sshd    6377      root    1   unknown                        /proc/6377/fd/1 (readlink: Permission denied)
sshd    6377      root    2   unknown                        /proc/6377/fd/2 (readlink: Permission denied)
sshd    6377      root    3   unknown                        /proc/6377/fd/3 (readlink: Permission denied)
sshd    6377      root    4   unknown                        /proc/6377/fd/4 (readlink: Permission denied)
sshd    6442      root  cwd   unknown                        /proc/6442/cwd (readlink: Permission denied)
sshd    6442      root  rtd   unknown                        /proc/6442/root (readlink: Permission denied)
sshd    6442      root  txt   unknown                        /proc/6442/exe (readlink: Permission denied)
sshd    6442      root    0   unknown                        /proc/6442/fd/0 (readlink: Permission denied)
sshd    6442      root    1   unknown                        /proc/6442/fd/1 (readlink: Permission denied)
sshd    6442      root    2   unknown                        /proc/6442/fd/2 (readlink: Permission denied)
sshd    6442      root    3   unknown                        /proc/6442/fd/3 (readlink: Permission denied)
sshd    6442      root    4   unknown                        /proc/6442/fd/4 (readlink: Permission denied)
sshd    6442      root    5   unknown                        /proc/6442/fd/5 (readlink: Permission denied)

There is some more output for the connected user, which is correctly identified as well, but that's it. It's apparently impossible to discern of which type (lsof -i limits to internet sockets) a certain "object" is.

0xC0000022L
  • 16,189
  • 24
  • 102
  • 168
  • What does a `lsof` report? The same? – slm Jun 12 '14 at 01:10
  • @slm: brilliant inquiry! It doesn't keep it empty. Instead it shows a whole host of (also `sshd`-related) lines, some of which could be TCP sockets, as `TYPE` `unknown`. Peculiar. Appending the output to my question. – 0xC0000022L Jun 12 '14 at 01:17
  • If you run `strace -s 2000 -o lsof.log lsof -i` it will likely give you some additional insights into what's getting blocked. – slm Jun 12 '14 at 01:45
  • 1
    @slm: good point. Thanks for reminding me. I'll do this tomorrow, though. Also well possible that `strace` itself is limited in the container. Exciting new stuff to learn. Thanks for bouncing idea. Must hit the bed, though. – 0xC0000022L Jun 12 '14 at 01:47
  • FYI: This also breaks netstat -lp. It is definitely caused by apparmor. – Alan Robertson Jul 08 '14 at 01:53

4 Answers4

11

(NOTE: it is unclear in the question how the asker is entering the docker container. I'm assuming docker exec -it CONTAINER bash was used.)

I had this problem when using a docker image based on centos:7 with docker version 1.9.0 and to overcome this, I just ran:

docker exec --privileged -it CONTAINER bash

Note the inclusion of --privileged.

My naive understanding of the reason this is required: it seems that docker makes an effort to have the container be more "secure", as documented here.

erik.weathers
  • 426
  • 5
  • 5
4

Hah, the plot thickens. If someone has a better answer please write it up and I'll accept it, if acceptable. But here the apparent reason. How negligent of me to ignore the log files on the host:

Jun 12 01:29:46 hostmachine kernel: [140235.718807] audit_printk_skb: 183 callbacks suppressed
Jun 12 01:29:46 hostmachine kernel: [140235.718810] type=1400 audit(1402536586.521:477): apparmor="DENIED" operation="ptrace" profile="docker-default" pid=3782 comm="lsof" requested_mask="trace" denied_mask="trace" peer="docker-default"
Jun 12 01:29:46 hostmachine kernel: [140235.718860] type=1400 audit(1402536586.521:478): apparmor="DENIED" operation="ptrace" profile="docker-default" pid=3782 comm="lsof" requested_mask="read" denied_mask="read" peer="docker-default"
Jun 12 01:29:46 hostmachine kernel: [140235.718886] type=1400 audit(1402536586.521:479): apparmor="DENIED" operation="ptrace" profile="docker-default" pid=3782 comm="lsof" requested_mask="read" denied_mask="read" peer="docker-default"
Jun 12 01:29:46 hostmachine kernel: [140235.718899] type=1400 audit(1402536586.521:480): apparmor="DENIED" operation="ptrace" profile="docker-default" pid=3782 comm="lsof" requested_mask="read" denied_mask="read" peer="docker-default"
Jun 12 01:29:46 hostmachine kernel: [140235.718921] type=1400 audit(1402536586.521:481): apparmor="DENIED" operation="ptrace" profile="docker-default" pid=3782 comm="lsof" requested_mask="read" denied_mask="read" peer="docker-default"
Jun 12 01:29:46 hostmachine kernel: [140235.718954] type=1400 audit(1402536586.521:482): apparmor="DENIED" operation="ptrace" profile="docker-default" pid=3782 comm="lsof" requested_mask="read" denied_mask="read" peer="docker-default"
Jun 12 01:29:46 hostmachine kernel: [140235.719001] type=1400 audit(1402536586.521:483): apparmor="DENIED" operation="ptrace" profile="docker-default" pid=3782 comm="lsof" requested_mask="read" denied_mask="read" peer="docker-default"
Jun 12 01:29:46 hostmachine kernel: [140235.719043] type=1400 audit(1402536586.521:484): apparmor="DENIED" operation="ptrace" profile="docker-default" pid=3782 comm="lsof" requested_mask="read" denied_mask="read" peer="docker-default"
Jun 12 01:29:46 hostmachine kernel: [140235.719086] type=1400 audit(1402536586.521:485): apparmor="DENIED" operation="ptrace" profile="docker-default" pid=3782 comm="lsof" requested_mask="read" denied_mask="read" peer="docker-default"
Jun 12 01:29:46 hostmachine kernel: [140235.719126] type=1400 audit(1402536586.521:486): apparmor="DENIED" operation="ptrace" profile="docker-default" pid=3782 comm="lsof" requested_mask="read" denied_mask="read" peer="docker-default"

So apparmor appears to be the culprit, although I'll have to figure out how to convince it to allow this without compromising host/container security or to see whether it's at all possible without compromising security.

0xC0000022L
  • 16,189
  • 24
  • 102
  • 168
4

Another possibility, this time with a more finegrained security setting: give the SYS_PTRACE privilege to the docker container:

docker run --cap-add=SYS_PTRACE ...
peterh
  • 9,488
  • 16
  • 59
  • 88
  • 1
    In case anyone's wondering why `lsof` needs `CAP_SYS_PTRACE`: It's required to read `/proc/*/stat`. See [ptrace(2)](http://man7.org/linux/man-pages/man2/ptrace.2.html) – David Röthlisberger Aug 22 '17 at 12:56
2

I have found this problem too. The problem has gone after I disabled apparmor on docker:

$ sudo aa-complain <docker apparmor profile name, "docker-default" on ubuntu>

reference url: https://help.ubuntu.com/community/AppArmor

menghan
  • 121
  • 4
  • 3
    You may want to consider adding more explanation to this answer (e.g., what `aa-complain` does, or some documentation which supports this solution). – HalosGhost Jul 08 '14 at 06:29
  • @HalosGhost Sorry, I'm not quite familiar with `apparmor`, I just googled it and found way to disable it. In other words, I don't know why it works or why it didn't work. My host os is Ubuntu 14.04, so I searched "ubuntu apparmor" and found https://help.ubuntu.com/community/AppArmor. I hope this would be helpful for you. – menghan Jul 08 '14 at 08:53
  • 2
    I do not have this issue; my concern was for the quality of your answer and how helpful (and informative) it would be for the OP. – HalosGhost Jul 08 '14 at 08:54
  • @HalosGhost Thanks for your help, I reedit my answer. – menghan Jul 08 '14 at 09:00
  • On ubuntu 14.04 the command was `sudo aa-complain /etc/apparmor.d/docker`. Basically it disables app armor for docker process which means that docker can read any file on the system. Previously it could only work with files that were allowed in the profile. A better solution might be to change the app armor rules that would allow access to the /proc/pid/fd files. – Martins Balodis Jan 28 '15 at 11:53