2

I send TCP data between hosts (the topology is created using network namespaces and veth-pairs). For a host, I save its ingress and egress packets to pcap-file using NFLOG and tcpdump with the following commands executed at the host:

# we turn off checksum offload:
sudo ethtool -K veth0 tx off sg off tso off ufo off

# we log packets with nflog:
sudo iptables -A OUTPUT -j NFLOG --nflog-group 17
sudo iptables -A INPUT  -j NFLOG --nflog-group 17

# we write the packets:
sudo tcpdump -i nflog:17 -w mypcap.pcap

So for all the outgoing TCP packets with zero Len the checksum is always wrong. This is true for all the hosts in the topology for egress traffic. For incoming traffic there is no such problem. This is because, as I checked (by regularly capturing with tcpdump over the host's interface rather than over NFLOG), when the egress traffic is leaving the host's interface the checksum is already corrected.

Pcap at sender (11.0.0.5), captured with NLOG:

enter image description here

Pcap at sender (11.0.0.5), captured regularly at sender's interface:

enter image description here

Pcap at receiver (11.0.0.1), captured with NLOG:

enter image description here

Pcap at receiver (11.0.0.1), captured regularly at receiver's interface:

enter image description here

So, as you can see in the images above, for pcaps captured from iptables NFLOG, for all the egress TCP pcakets which have Len equal to zero, TCP checksum is wrong. What can be the reason?

Thank you for attention!

JenyaKh
  • 304
  • 4
  • 14

1 Answers1

1

I have found the solution to the problem. It is possible to use iptables NFQUEUE instead of NFLOG. Besides solving the problem with TCP checksums, the advantage of this method is also that the packets get captured without “Linux Netfilter NFLOG” Link-Layer header, that is, the packets in the resulting pcap-file are just raw-ip packets.

sudo iptables -A INPUT  -p tcp -s $receiver-ip -d $sender-ip   -j NFQUEUE --queue-num 17
sudo iptables -A OUTPUT -p tcp -s $sender-ip   -d $receiver-ip -j NFQUEUE --queue-num 17
sudo iptables -A INPUT  -p udp -s $receiver-ip -d $sender-ip   -j NFQUEUE --queue-num 17
sudo iptables -A OUTPUT -p udp -s $sender-ip   -d $receiver-ip -j NFQUEUE --queue-num 17

sudo tcpdump -i nfqueue:17 -w mypcap.pcap

I was searching for the solution for quite long to solve this problem: Tc qdisc delay not seen in tcpdump recording. First, the solution looked quite good to me: no Link-Layer headers which made dump files smaller, TCP checksum problem cured. But it turned out that with big sending rates (3Gbps when no netem rate/delay is installed at veth-pair links of the topology) only half all the traffic gets recorded. That is, if I record ingress and egress traffic at sender capturing packets at interface I get e.g. dump of 1.7 Gigabytes, while if I record the traffic capturing from kernel's iptables the dump is about 900 Megabytes. This problem with sending rate getting throttled happens both for NFLOG and NFQUEUE solutions.

JenyaKh
  • 304
  • 4
  • 14