11

Network monitoring tools ss, lsof -i, netstat -ltupw and similar have limitations like namespace and thus they don't show the whole machine connections. (for namespace created with ip we can see other connections with ip -all netns exec ss).

The kernel state the connections here /proc/net/tcp[udp][raw] and namespace connections can be seen here /proc/$pid/net/tcp[udp][raw] (detail)

What are the other possible connections that are not seen by ss/lsof/netstat and how to monitor them?

intika
  • 13,920
  • 7
  • 41
  • 79
  • 2
    I guess that a start of an answer would be 1. connection outside the kernel like Intel me 2. namespaces 3. other kernel modules that would use a particular interface... – intika Jun 03 '19 at 13:03
  • 3
    Any connection created via raw sockets? the sockets will appear in the info obtained with sock_diag (as used by ss, etc), but not the "connections" an app may build on top of them. Your question is too open-ended, so I doubt that any adequate answer can be made. The only true way is to capture all the packets on the wire and look at them. –  Jun 03 '19 at 13:22
  • Raw ip packets and raw ethernet packets both use the kernel socket and thus are visible to the listed tools, indeed an application can build its own protocol on top of a raw socket but nevertheless the raw connection is visible. i don't think the question is broad it ask for what is not visible for the usual monitoring tool. – intika Jun 03 '19 at 15:53
  • 2
    @intika You likely know more than me but maybe this is helpful: 1) `NFQUEUE` can reveal connections as they happen, but it might be tricky to set up and not miss anything (`OpenSnitch` has a number of closed issues covering that part, although I'm not sure if there's anything about Layer2 there) 2) I'm guessing that kernel extensions are able to make unnoticed connections if they talk directly to NIC (not directly applicable on Linux, but I'm observing something like this on a Mac, where VirtualBox is able to bypass Little Snitch when in bridged mode). – Tomasz Zieliński Jun 03 '19 at 16:28
  • 4
    Very fast connections - those are not going to be visible. If you're doing something like `watch -n1 ss` you won't see connections established and closed within less than 1 second. – Sergiy Kolodyazhnyy Jun 03 '19 at 18:12
  • 1
    @intika "Raw ip packets and raw ethernet packets both use the kernel socket and thus are visible to the listed tools" Neither ss, nor lsof, nor netstat knows anything abouts packets, raw or cooked. They only display info about sockets, gathered from ioctls, /proc files and (more recently) from the *diag netlink interface. Knowing that process X has a raw socket open won't tell you much about what it's using it for, unlike a tcp socket, which has a peer, a port, etc. –  Jun 03 '19 at 18:42
  • I came across that putting the network device in promiscuous mode will get all the traffic that the network card receives. Bear in mind that if you machine is acting as a router traffic that is forwarded is not tracked by the kernel and you will some tool for see the status of that connections. – camp0 Jun 07 '19 at 07:40
  • 1
    non IP based connection. you have other network protocosl, ss, netstat are basically TCP/UDP/other IP tools, lsof i think only looks for tcp/ip. the most common protocol is ARP, the system is always using it and it is invisible for most tools. tcpdump is your friend, even if it have tcp in the name, it too can detect other protocols – higuita Feb 10 '22 at 23:04
  • @SergiyKolodyazhnyy with kernel instrumentation, things (except the instrumentation itself...) become easier. For example [libbpf-tools](https://github.com/iovisor/bcc/tree/master/libbpf-tools)'s `tcpstates` example tool run from the host will display TCP socket state activity in any network namespace, including process name and id. – A.B Apr 16 '23 at 14:51

0 Answers0