7

A program injects packets on a Linux TAP interface (these packets are coming from a virtual machine). Specifically, these are DHCP requests (so they're UDP). I can see the packets with tcpdump but not with iptables, and they don't reach the local DHCP server either. Why not, and how do I fix that?

Update: I tried injecting IP packets directed to the address of the tap0 interface. I see the ARP requests coming in from the VM in tcpdump -i tap0, but the network layer does not reply. If I send ARP requests to the VM, it sees them and replies to the host (and the replies show up in tcpdump but are otherwise lost).

Another observation: ifconfig tap0 shows that the TX dropped packet count is incremented for each packet that's injected onto the host. Why TX?

# ifconfig tap0
…
          TX packets:0 errors:0 dropped:958 overruns:0 carrier:0
          collisions:0 txqueuelen:500
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

The long story: On a Linux host (running Ubuntu 10.04), I'm running a virtual machine which amongst other things emulates an Ethernet card. It does so by communicating with a helper program that's in charge of injecting and capturing Ethernet packets onto the hosts's network stack. The virtual machine is an ARM chip emulator, and the helper program is called nicserver; all I know about it is what is found in the ARM documentation.

I want to establish an Ethernet link between the VM and the host, and above that I want an IP link. The VM obtains its IP address over DHCP. I do not want any communication between the VM and the rest of the world, only with the host, so I created a virtual network interface tap0 with

tunctl -u gilles
ifconfig tap0 192.168.56.1 netmask 255.255.255.0 up
nicserver -p 7801 -a tap0 &

Now I boot the VM, and I can see it's sending DHCP requests with tcpdump -n -i tap0 -vv (the DHCP client doesn't time out, I'm just showing one sample request here):

tcpdump: listening on tap0, link-type EN10MB (Ethernet), capture size 96 bytes
18:29:23.941574 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto UDP (17), length 576)
    0.0.0.0.68 > 255.255.255.255.67: [no cksum] BOOTP/DHCP, Request from 02:52:56:47:50:03, length 548, xid 0x238a7979, secs 46, Flags [none] (0x0000)
          Client-Ethernet-Address 02:52:56:47:50:03 [|bootp]

I've set up Dnsmasq on the host to serve the requests, but it's not seeing any incoming request. The Dnsmasq server doesn't even see the incoming requests (I straced it). So I tried observing the packets with Iptables. (All the filter/INPUT rules are shown; there are no mangle or nat rules).

Chain INPUT (policy ACCEPT 2366K packets, 5334M bytes)
 pkts bytes target     prot opt in     out     source               destination 
  119 39176 LOG        udp  --  *      *       0.0.0.0/0            0.0.0.0/0           udp dpt:67 LOG flags 4 level 4 prefix `[DHCP request] '
  119 39176 DROP       udp  --  eth1   *       0.0.0.0/0            0.0.0.0/0           udp dpt:67
    2   490 LOG        udp  --  tap0   *       0.0.0.0/0            0.0.0.0/0           LOG flags 4 level 4 prefix `[in=tap0] '
   26  6370 ACCEPT     udp  --  tap0   *       0.0.0.0/0            0.0.0.0/0   
    0     0 ACCEPT     all  --  tap0   *       0.0.0.0/0            0.0.0.0/0   
 3864  457K ACCEPT     udp  --  *      *       0.0.0.0/0            0.0.0.0/0

All these incoming DHCP requests are on eth1 (and I'm careful not to ignore these so as not to anger my colleagues and my network admin). Those UDP packets on tap0 come from the local Samba server. The DHCP request packets that I see with tcpdump do not appear to go through the packet filter!

Why do I see incoming broadcast packets on tap0 with tcpdump but not with iptables (nor with programs listening on the machine)? And what do I need to fix so that these packets are seen, as they would be if they were coming on an Ethernet interface?

Gilles 'SO- stop being evil'
  • 807,993
  • 194
  • 1,674
  • 2,175
  • maybe try using physdev matching iptables and see if that changes how you're seeing the traffic. `iptables -A INPUT -p udp --dport 67 -m physdev --physdev-in tap+ -j LOG --log-level 4 --log-prefix "[in=tap0] "` – Tim Kennedy Nov 08 '11 at 19:51
  • `-m physdev` is usually used with bridges and tap interfaces. i know you're not trying to bridge the tap interface to your eth, but I'm wondering if using the physdev to match the interface somehow changes the traffic to make it visible? I have no idea if it will work, or change anything, though. – Tim Kennedy Nov 08 '11 at 20:22
  • @TimKennedy doesn't change anything. More oddness: if I tell `nicserver` to use `eth0`, even `tcpdump` doesn't see any packets. This could be an issue with `nicserver`, but I still want to know how packets can be seen by `tcpdump` on `tap0` and not by `iptables`. – Gilles 'SO- stop being evil' Nov 09 '11 at 11:39
  • 1
    if you run `tcpdump` with `-p`, can `tcpdump` still see the packets? (-p restricts `tcpdump` from putting the interface in promiscuous mode.) Since `iptables` doesn't put the interface into promiscuous mode, you'll get a closer comparison to what `iptables` sees. – Tim Kennedy Nov 09 '11 at 12:27
  • @TimKennedy `nicserver` puts the interface in promiscuous mode permanently (`ifconfig` confirms this). I notice something peculiar: each “lost” DHCP request increments the TX dropped counter by 1. Could it be that the packets are somehow injected in a way that makes the kernel consider them as sent and not received? I tried logging UDP output, but the requests don't appear in the filter/OUTPUT chain either. – Gilles 'SO- stop being evil' Nov 09 '11 at 17:33
  • Maybe try netstat -s tap0 to look for more clues as to why the packets are being dropped (what other counters are incrementing). – shank Nov 10 '11 at 17:57
  • @shank `netstat -s` is global, and I can't stop unrelated traffic on `eth1. In `/sys/class/net/tap0/statistics`, only `tx_dropped` is nonzero. Anything else I should look at? (Note: further tests will have to wait until Monday.) – Gilles 'SO- stop being evil' Nov 10 '11 at 18:25
  • Are you sure the tap0 packets enter the input chain? They could as well be subject to routing. – wnrph Nov 13 '11 at 09:25
  • @artistoex I can see they don't enter the input chain, and I'm trying to find out why. I know I have no set any routing table entries beyond the one implied by bringing up an interface with a netmask, plus a default route through `eth1`. – Gilles 'SO- stop being evil' Nov 13 '11 at 13:25
  • On a router no packet passing it would enter the input chain, but each would show up in a tcpdump. Maybe TAP traffic is likewise regarded as routed traffic? Another thought: Non-IP packets might not be handled at all by iptables (one could use ebtables, instead). – wnrph Nov 14 '11 at 07:51
  • @artistoex What should I be looking at or doing then? I'm fairly sure there was nothing exotic about that machine's networking setup until I created the tap device. – Gilles 'SO- stop being evil' Nov 14 '11 at 08:29
  • 2
    Insert your log target into the prerouting chain. I also recommend the nice [iptables flowchart](http://upload.wikimedia.org/wikipedia/commons/3/37/Netfilter-packet-flow.svg) on wikipedia. – wnrph Nov 14 '11 at 12:29
  • @artistoex Thanks. No IP packet from the VM appears in the iptables raw/PREROUTING, nat/PREROUTING, mangle/PREROUTING or mangle/INPUT. No Ethernet packet from the VM's MAC appears in the ebtables broute/BROUTING, nat/PREROUTING or filter/INPUT. – Gilles 'SO- stop being evil' Nov 14 '11 at 15:38
  • It's been some time since you've posted this question. Have you had any success so far? – wnrph Nov 21 '11 at 10:54
  • @artistoex A year later I came back to this and realized that I see no traffic with iptables because the first thing the VM does is send an ARP request which the host doesn't see. With no ARP response, there is no IP traffic for iptables to see. Duh. But that doesn't explain the root problem, which is that the host isn't communicating with the VM. I still don't understand what's going on, and I suspect a bug in `nicserver`. This is a very old utility, and we're running an unsupported version of Linux (far too recent and the wrong distribution to boot). I lack the time to investigate further. – Gilles 'SO- stop being evil' Nov 07 '12 at 19:00

2 Answers2

1

Here is further guesswork. Hope this is helpful, but it may as well be embarassingly wrong.

tap0 has two ends, the kernel network stack end and the program interface. It seems to me if you supply 'nicserver' with tap0, it won't attach to it the way it is intended with tap devices, using the program interface. Instead, nicserver will simply write to it from the network stack end, and with no application reading from the program interface end, you will end up overflowing the device queue. This explains the dropped packets. Also, no packets will be delivered, which might explain the iptables result.

I guess if you let tcpdump capture on tap0 it actually attaches to the program interface end of tap0 and, voila, you see the packets. I searched the internet but found no source to confirm such behaviour. To falsify this theory, capture tap0 and see how it affects packet drops and the iptables log.

Finally, an advice that addresses your original problem: what about using the loopback device, instead? Like so: 

nicserver -p 7801 -a lo
wnrph
  • 1,414
  • 13
  • 16
0

The iptable table you are printing is the filter table.

The filter table is checked for match only after a routing decision is made.

Check the figure in this blog: https://www.garron.me/en/linux/iptables-manual.html

Check if the other tables show packet count increase when you pass traffic.

iptables -nvL -t raw;
iptables -nvL -t nat;
iptables -nvL -t mangle;
FargolK
  • 1,629
  • 1
  • 12
  • 20
Haswell
  • 131
  • 6