5

I have two linux containers connected with a veth-pair. At veth-interface of one container I set up tc qdisc netem delay and send traffic from it to the other container. If I watch traffic on both sides using tcpdump/wireshark it can be seen that timestamps of the same packet at sender and receiver do not differ by selected delay.

I wanted to understand more in detail at which point libpcap puts timestamps to egress traffic corresponding to tc qdisc. I searched for a scheme/image on Internet but did not find. I found this topic (wireshark packet capture point) but it advises to introduce an indirection by having one more container/interface. This is not a possible solution in my situation. Is there any way to solve the problem not introducing additional intermediate interfaces (that is, not changing topology) and only by recording at the already given veth-interface but in such a way that the delay can be seen?

UPDATE:

I was too quick and so got mistaken. Neither my solution present below (same as the first variant of solution of the answer of @A.B), nor the solution with IFB of @A.B (I have already checked) solve my problem. The problem is with overflow of transmit queue of interface a1-eth0 of sender in the topology:

[a1-br0 ---3Gbps---a1-eth0]---100Mbps---r1---100Mbps---r2

I was too quick and checked only for delay 10ms at link between a1-eth0 and router r1. Today I tried to make the delay higher: 100ms, 200ms and the results (per-packet delay and rate graphs which I get) start to differ for the topology above and for the normal topology:

[a1-eth0]---100Mbps---r1---100Mbps---r2

So no, certainly, for accurate testing I cannot have extra links: nor introduced by Linux bridge, nor by this IFB, nor by any other third system. I test congestion control schemes. And I want to do it in a specific topology. And I cannot change the topology just for the sake of plotting -- I mean if at the same time my rate and delay results/plots get changed.

UPDATE 2:

So it looks like the solution has been found, as it can be seen below (NFLOG solution).

UPDATE 3:

Here are described some disadvantages of NFLOG solution (big Link-Layer headers and wrong TCP checksums for egress TCP packets with zero payload) and proposed a better solution with NFQUEUE which does not have any of these problems: TCP checksum wrong for zero length egress packets (captured with iptables). However, for my tasks (testing of congestion control schemes) neither NFLOG, nor NFQUEUE are suitable. As it is explained by the same link, sending rate gets throttled when packets get captured from kernel's iptables (this is how I understand it). So when you record at sender by capturing from interface (i.e., regularly) you get 2 Gigabytes dump, while if you record at sender by capturing from iptables you get 1 Gigabyte dump. Roughly speaking.

UPDATE 4:

Finally, in my project I use Linux bridge solution described in my own answer bewow.

JenyaKh
  • 304
  • 4
  • 14

2 Answers2

6

According to the Packet flow in Netfilter and General Networking schematic, tcpdump captures (AF_PACKET) after egress (qdisc). So it's normal you don't see the delay in tcpdump: the delay was already present at initial capture.

You'd have to capture it one step earlier, so involve a 3rd system:

S1: system1, runs tcpdump on outgoing interface
R: router (or bridge, at your convenience, this changes nothing), runs the qdisc netem
S2: system2, runs tcpdump on incoming interface 

 __________________     ________________     __________________
|                  |   |                |   |                  |
| (S1) -- tcpdump -+---+- (R) -- netem -+---+- tcpdump -- (S2) |
|__________________|   |________________|   |__________________|

That means 3 network stacks involved, be they real, vm, network namespace (including ip netns, LXC, ...)

Optionally, It's also possible to cheat and move every special settings on the router (or bridge) by using an IFB interface with mirred traffic: allows by a trick (dedicated for this case) to insert netem sort-of-after ingress rather than on egress:

 _______     ______________________________________________     _______
|       |   |                                              |   |       |         
| (S1) -+---+- tcpdump -- ifb0 -- netem -- (R) -- tcpdump -+---+- (S2) |
|_______|   |______________________________________________|   |_______|

There's a basic IFB usage example in tc mirred manpage:

Using an ifb interface, it is possible to send ingress traffic through an instance of sfq:

# modprobe ifb
# ip link set ifb0 up
# tc qdisc add dev ifb0 root sfq
# tc qdisc add dev eth0 handle ffff: ingress
# tc filter add dev eth0 parent ffff: u32 \
  match u32 0 0 \
  action mirred egress redirect dev ifb0

Just use netem on ifb0 instead of sfq (and in non-initial network namespace, ip link add name ifbX type ifb works fine, without modprobe).

This still requires 3 network stacks for proper working.

using NFLOG

After a suggestion from JenyaKh, it turns out it's possible to capture a packet with tcpdump, before egress (thus before qdisc) and then on egress (after qdisc): by using iptables (or nftables) to log full packets to the netlink log infrastructure, and still reading them with tcpdump, then again using tcpdump on the egress interface. This requires only settings on S1 (and doesn't need a router/bridge anymore).

So with iptables on S1, something like:

iptables -A OUTPUT -o eth0 -j NFLOG --nflog-group 1

Specific filters should probably be added to match the test done, because tcpdump filter is limited on nflog interface (wireshark should handle it better).

If the answer capture is needed (here done in a different group, thus requiring an additional tcpdump):

iptables -A INPUT -i eth0 -j NFLOG --nflog-group 2

Depending on needs it's also possible to move them to raw/OUTPUT and raw/PREROUTING instead.

With tcpdump:

# tcpdump -i nflog:1 -n -tt ...

If a different group (= 2) was used for input:

# tcpdump -i nflog:2 -n -tt ...

Then at the same time, as usual:

# tcpdump -i eth0 -n -tt ...
A.B
  • 31,762
  • 2
  • 62
  • 101
  • Thank you very much for your answer! Yesterday, I came up with another solution (unexpectedly using Linux bridge -- the solution which I had previously considered as a bad one, as I explain in my answer). Anyway, I really appreciate your help. As soon as I get enough reputation, I will not forget to upvote your answer! – JenyaKh Jun 28 '19 at 17:48
  • I have read your answer again and realized that the first part of it is close to mine. Also, you provide a great explanation+image of where Wireshark captures stuff. So as you were the first, I accept your answer -- this is fair. Thank you again! – JenyaKh Jun 28 '19 at 18:45
  • You know I've made an update to my question. Neither your nor mine solutions worked out for me. The problem is with transmit queue overflow described in my answer (wrong answer it turned out). I explain in my update why I cannot use third systems in my topology. Maybe, do you have any ideas on the situation? Would be very grateful for your comment. – JenyaKh Jun 29 '19 at 20:59
  • Well you're changing the question but at the same time you stated you didn't want a 3rd thing. I still explained why tcpdump didn't work. Feel free to un-accept my answer. 2 comments to follow for alternative ways, requiring work (development, tool tweaking etc.) – A.B Jun 29 '19 at 22:03
  • If it's not possible to use the time when measuring,it should be possible to use the tools generating the test directly.Eg: iperf3 mentions in 2.0.7 "Use linux realtime scheduler and **packet level timestamps** for improved latency accuracy". IP TS option has a 1ms resolution. TCP TS option is (not sure) also 1ms and 64bits. So if your systems are synchronized you can still capture the time. Be sure to not hardware offload it.You might have to create your own tools. – A.B Jun 29 '19 at 22:03
  • If this still inserts the stamp only at egress this won't help. I think there are options there: https://www.kernel.org/doc/Documentation/networking/timestamping.txt (good candidate: SOF_TIMESTAMPING_TX_SCHED). Else you can send time informations as data. That's probably what is doing the NTP protocol to synchronize time over arbitrary network delays. – A.B Jun 29 '19 at 22:03
  • I see, thank you! Can I ask you on what do you think of Ulogd variant? Can it work out? – JenyaKh Jun 29 '19 at 22:10
  • 1
    ulogd (d as daemon) itself maybe not (latency...),but sending packets to the netlink log infrastructure using iptables **NFLOG** target is a good idea.tcpdump can listen on netlink logs as if it was a standard interface (but I don't think it could distinguish different --nflog-group, so you're stuck with only one nflog capture per system, or you need an other tool than tcpdump. libpcap knows about it though).try `tcpdump -D` to get the **nflog** interface index, then you can capture with -i *index* instead of a standard interface name. You could then capture on the egress interface and compare – A.B Jun 29 '19 at 22:20
  • Great! That worked awesome. I have just captured TCP dst 80 traffic on my own machine with NFLOG+tcpdump. That's awesome! I will check if this also works for my topology and if this finally solves my delay measuring problem. Thank you! – JenyaKh Jun 29 '19 at 22:30
1

UPDATE:

So I finally used this solution. It is present in my solution. It worked well for me after all.

I (the topic starter) have solved my problem using Linux bridge. Here [https://www.linuxquestions.org/questions/linux-networking-3/transferring-all-traffic-through-an-extra-interface-4175656515 ] I wrote that I managed to use Linux bridge but dismissed the possibility: "But this solution does not suit my needs, as there is an extra Ethernet link between h1-br0 and h1-eth0 interfaces in reality. I need this stuff for performance measurements so I cannot have any extra Ethernet links. I mean this solution with bridge messes up my topology by introducing extra links."

       a1
-----------------
|a1-br0---a1-eth0|---------local network
------------------

Why did I dismiss the solution first? Initially, my topology is:

a1---3Gbps---r1---100Mbps---r2

On the link r1---r2 I have netem rate set to 100 Mbps, on the link a1---r1 there is no rate limits. As the transmit queue of router r1 connecting it to router r2 is 1000 packets I had the effect of queue overflow (some packets get dropped) when sending traffic from a1 to r2. And this was okay. This is how it happens in real world with router queues get overflowed in case of bottleneck link.

Now I do all this research to add delay and rate limits to a1---r1 also. So I came up with this solution using Linux bridge. But I thought that this solution will not work. Below you can see the new topology with Linux bridge:

[a1-br0 ---3Gbps---a1-eth0]---100Mbps---r1---100Mbps---r2

So my problem with the solution was that I expected the queue overflow would be present now at the transmit queue of interface a1-eth0. That is, this is the same way as in the previous picture where the overflow was at the interface of r1 connecting it to r2. Analogously.

And this overflow I do not want. Because in the normal topology -- without using Linux bridge for the purpose of delay measurement -- we do not have any overflow of transmit queue of a1-eth0:

[a1-eth0]---100Mbps---r1---100Mbps---r2

But yesterday I created the topology with Linux bridge (the 3rd topology of the drawn above) again and launched traffic at the topology flowing from a1 to r2. I checked the backlog (current number of packets in queue) of the transmit queue of a1-eth0 calling the command tc -s qdisc show dev a1-eth0 in cycle with 500ms interval and backlog of the transmit queue of a1-br0 with the analogous command.

This is what I saw for a1-eth0, I got the messages:

qdisc netem 8112: root refcnt 2 limit 1000 delay 10.0ms
 Sent 9461862 bytes 6393 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 133380b 90p requeues 0 

qdisc netem 8112: root refcnt 2 limit 1000 delay 10.0ms
 Sent 15280534 bytes 10323 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 133380b 90p requeues 0 

qdisc netem 8112: root refcnt 2 limit 1000 delay 10.0ms
 Sent 21110722 bytes 14257 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 118560b 80p requeues 0 

qdisc netem 8112: root refcnt 2 limit 1000 delay 10.0ms
 Sent 26952766 bytes 18199 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 102258b 69p requeues 0 

qdisc netem 8112: root refcnt 2 limit 1000 delay 10.0ms
 Sent 32788882 bytes 22137 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 103740b 70p requeues 0 

qdisc netem 8112: root refcnt 2 limit 1000 delay 10.0ms
 Sent 38635372 bytes 26082 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 102258b 69p requeues 0 

qdisc netem 8112: root refcnt 2 limit 1000 delay 10.0ms
 Sent 44477416 bytes 30024 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 102258b 69p requeues 0 

qdisc netem 8112: root refcnt 2 limit 1000 delay 10.0ms
 Sent 50332798 bytes 33975 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 102258b 69p requeues 0 

qdisc netem 8112: root refcnt 2 limit 1000 delay 10.0ms
 Sent 56157058 bytes 37905 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 125970b 85p requeues 0 

qdisc netem 8112: root refcnt 2 limit 1000 delay 10.0ms
 Sent 61969532 bytes 41828 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 133380b 90p requeues 0 

qdisc netem 8112: root refcnt 2 limit 1000 delay 10.0ms
 Sent 67784900 bytes 45752 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 133380b 90p requeues 0 

qdisc netem 8112: root refcnt 2 limit 1000 delay 10.0ms
 Sent 73600268 bytes 49676 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 133380b 90p requeues 0 

qdisc netem 8112: root refcnt 2 limit 1000 delay 10.0ms
 Sent 79415636 bytes 53600 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 133380b 90p requeues 0 

qdisc netem 8112: root refcnt 2 limit 1000 delay 10.0ms
 Sent 85244342 bytes 57533 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 120042b 81p requeues 0 

qdisc netem 8112: root refcnt 2 limit 1000 delay 10.0ms
 Sent 91080458 bytes 61471 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 102258b 69p requeues 0 

qdisc netem 8112: root refcnt 2 limit 1000 delay 10.0ms
 Sent 96923984 bytes 65414 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 102258b 69p requeues 0 

qdisc netem 8112: root refcnt 2 limit 1000 delay 10.0ms
 Sent 102761582 bytes 69353 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 102258b 69p requeues 0 

qdisc netem 8112: root refcnt 2 limit 1000 delay 10.0ms
 Sent 108606590 bytes 73297 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 103740b 70p requeues 0

This is what I saw for a1-br0, I got the messages:

qdisc noqueue 0: root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0 

qdisc noqueue 0: root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0 

qdisc noqueue 0: root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0 

qdisc noqueue 0: root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0 

qdisc noqueue 0: root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0 

qdisc noqueue 0: root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0 

qdisc noqueue 0: root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0 

qdisc noqueue 0: root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0 

qdisc noqueue 0: root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0 

qdisc noqueue 0: root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0 

qdisc noqueue 0: root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0 

qdisc noqueue 0: root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0 

qdisc noqueue 0: root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0 

qdisc noqueue 0: root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0 

qdisc noqueue 0: root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0 

qdisc noqueue 0: root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0 

qdisc noqueue 0: root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0 

qdisc noqueue 0: root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0

So it can be seen that no overflow happens at a1-eth0 and in reality it does not "look" like a1-br0 sends anything, though in reality it sends. So the link between a1-bro and a1-eth0 is not like that one (the veth pair link) between a1 and router r1. I do not know why it is so. It is strange because I checked that I can for example set netem delay setting at a1-br0 -- so it is like a normal interface.

Anyway, I checked that the solution with the bridge satisfies all my needs. I have not yet explored why it works though (I mean in the sense of what I was explaining above -- queue overflow and etc.).

Here is the commands which I ran at the host a1 for reference. I understand that it is difficult to fully understand them without the context, though. But, maybe, it will help somebody in the future:

brctl addbr a1-br0
brctl addif a1-br0 a1-eth0
ip link set dev a1-br0 up
ip addr add dev a1-br0 11.0.0.1/30
ip addr flush dev a1-eth0
route add default gw 11.0.0.2 dev a1-br0
ifconfig a1-eth0 0.0.0.0 up
ethtool -K a1-br0 tx off sg off tso off ufo off

The topology with IP-addresses, to which I applied the commands, is also present here: Pinging one interface of Linux router by another interface of this router. Here is the topology:

------                           ------                            ------
| a1 |                           | r1 |                            | r2 |
|    | a1-eth0-----------r1-eth0 |    |r1-eth1--------------r2-eth1|    |
-----(11.0.0.1/30)   (11.0.0.2/30)----(11.0.0.9/30)   (11.0.0.10/30)-----
JenyaKh
  • 304
  • 4
  • 14