9

I tried both netstat and lsof, but it appears it's not possible to see the connections to my LXC guests.

Is there a way to achieve this ... for all guests at once?

Essentially what throws me off here is the fact that I can see the processes of the guests as long as I run as superuser. I can also see the veth interfaces that get dynamically created per guest. Why can I not see connections on processes that are otherwise visible?

0xC0000022L
  • 16,189
  • 24
  • 102
  • 168

5 Answers5

15

The kernel indicate the connections state on /proc/net/tcp, /proc/net/udp etc. but as the namespaces separate the network stack if an application is running inside a container (a different userspace) and is connected to the network the host /proc/net/tcp won't show its connection,

conntrack can be used to show the whole machine connection but this does not work for some interfaces like wireguard...

ip -all netns exec command can be used to run commands inside all the userspaces but this is limited to userspaces created with ip command.

On the perspective of an application running on a container its network stack state is still visible on the host on the location /proc/$pid/net/tcp so as a workaround awaiting to write a proper tool in c, i wrote a little bash script that loop on /proc/$pid/net/tcp[udp] and join all the states to be able to list the whole machine connection.

The script first join all /proc/$pid/net/tcp or /proc/$pid/net/udp sort them, remove duplicate, translate the value to a readable text and print them (the script require find, grep, xargs, awk, strtonum, sort and uniq)

For TCP

find /proc/ 2>/dev/null | grep tcp | grep -v task | grep -v sys/net | xargs grep -v rem_address 2>/dev/null | awk '{x=strtonum("0x"substr($3,index($3,":")-2,2)); y=strtonum("0x"substr($4,index($4,":")-2,2)); for (i=5; i>0; i-=2) x = x"."strtonum("0x"substr($3,i,2)); for (i=5; i>0; i-=2) y = y"."strtonum("0x"substr($4,i,2))}{printf ("%s\t:%s\t ----> \t %s\t:%s\t%s\n",x,strtonum("0x"substr($3,index($3,":")+1,4)),y,strtonum("0x"substr($4,index($4,":")+1,4)),$1)}' | sort | uniq --check-chars=25

For UDP

find /proc/ 2>/dev/null | grep udp | grep -v task | grep -v sys/net | xargs grep -v rem_address 2>/dev/null | awk '{x=strtonum("0x"substr($3,index($3,":")-2,2)); y=strtonum("0x"substr($4,index($4,":")-2,2)); for (i=5; i>0; i-=2) x = x"."strtonum("0x"substr($3,i,2)); for (i=5; i>0; i-=2) y = y"."strtonum("0x"substr($4,i,2))}{printf ("%s\t:%s\t ----> \t %s\t:%s\t%s\n",x,strtonum("0x"substr($3,index($3,":")+1,4)),y,strtonum("0x"substr($4,index($4,":")+1,4)),$1)}' | sort | uniq --check-chars=25

The output look like: (note that the pid is not accurate and is just used to identify the container) 

127.0.0.1      :80       ---->   0.0.0.0        :0      /proc/10176/net/tcp:
192.168.0.2    :33882    ---->   192.30.253.125 :443    /proc/10176/net/tcp
192.168.0.2    :34020    ---->   192.30.253.125 :443    /proc/10176/net/tcp:
192.168.0.2    :34162    ---->   192.30.253.125 :443    /proc/10176/net/tcp:
192.168.0.2    :36242    ---->   192.30.253.124 :443    /proc/10176/net/tcp:
192.168.0.2    :37324    ---->   192.30.253.124 :443    /proc/10176/net/tcp:
192.168.0.2    :40122    ---->   216.239.38.21  :80     /proc/10176/net/tcp:
192.168.0.2    :40124    ---->   216.239.38.21  :80     /proc/10176/net/tcp:

Also i found a great tool for managing namespace with some very useful commands that is nsutils

intika
  • 13,920
  • 7
  • 41
  • 79
2

I think https://stackoverflow.com/a/40352004/1951468 should answer you.

So basically, the only way is a loop around your containers and nsenter.

Silex
  • 121
  • 3
1

I think you can use sudo conntrack -L

conntrack is a userspace command line program targeted at system administrators. It enables them to view and manage the in-kernel connection tracking state table.

If not installed, the required package is conntrack-tools (on Fedora etc.) or conntrack (on Debian, Ubuntu etc.).

Stephen Kitt
  • 411,918
  • 54
  • 1,065
  • 1,164
suside
  • 237
  • 2
  • 5
  • Conntrack could also be queried by parsing `/proc/net/ip_conntrack`, which is effectively what your suggested program does. But both `lsof` and `netstat` use kernel information, for that matter, so what's the difference? – 0xC0000022L Mar 04 '19 at 20:29
  • 1
    If run from host `conntrack` shows connections established inside containers. Both `netstat` and `lsof` does not show that info. – suside Mar 06 '19 at 07:09
  • This does not work for me, it show the connections but not the namespaces connections... – intika May 27 '19 at 23:59
  • @intika this is very strange, are you 100% sure there is active connection inside container? Can you leave `ping` running inside and verify again? Also in Ubuntu there is `conntrack` package not `conntrack-tools`. – suside May 28 '19 at 12:02
  • Yes, `ip -all netns exec ss` show those connections, and `lsmod | grep conntrack` show `nf_conntrack, xt_conntrack, nf_conntrack_ipv4 & nf_conntrack_* etc` also i dont have `/proc/net/ip_conntrack`; `conntrack -L` does show the host connections but not those established inside other namespaces, and finally running `conntrack -L` from within other namespaces return an empty list @suside – intika May 28 '19 at 12:15
  • @suside i guess this is happening because of the network interface i am using inside the namespace which is wireguard – intika May 28 '19 at 12:55
0

A similar workaround is to execute the netstat/lsof command on all network namespace with 

sudo ip -all netns exec ss -p -ut

or

sudo ip -all netns exec lsof -i

or

sudo ip -all netns exec netstat -ltup
intika
  • 13,920
  • 7
  • 41
  • 79
-2

So far I had luck getting the container connections by running netstat from the container:

sudo docker exec -it <containerIdOrName> netstat

Obviously, the container needs to have it installed.

AXE Labs
  • 265
  • 2
  • 3