4

I'm trying to modify the network behaviour of my server(s), to simulate external/WAN connection behaviours (what ever that means).

After doing tc qdisc add dev lo root netem delay 100ms, I can successfully add 100ms delay to all traffic from (and to?) 127.0.0.1. E.g. ping 127.0.0.1 will have 200ms response time.

However, this also affect my traffic to other interfaces. For example, I have interface eth1 with IP address 192.168.0.1 on the current server A. If I do ping 192.168.0.1, it will also have the 100ms delay (resulting in 200ms response time).

I'm confused by this behaviour. I would expect lo has nothing to do with eth1, but it seems not to be the case.

I assume this means Linux kernel automatically identifies 192.168.0.1 is a local address, and re-routes all traffic (originally to eth1) to lo?

And if so, is there a way to disable this behaviour?

Background:

I would like to simulate external network behaviour even when processes on server A want to communicate to each other (through TCP/IP on the given local addresses and ports, of course). Essentially I want to add delay to eth1, but that's above this question (see my other question).

My servers are running Ubuntu 18.04, but I believe that does not matter.

renyuneyun
  • 377
  • 2
  • 3
  • 11

1 Answers1

2

No it doesn't affect other interfaces. But the routing involved makes that any access from the server to itself stays local and uses the lo (loopback) interface whatever interface the IP address was assigned to. So lo is affected by tc ... netem.

You can verify this with the ip route get command, which will give something similar to this for your case:

$ ip route get 192.168.0.1
local 192.168.0.1 dev lo table local src 192.168.0.1 uid 1000 
    cache <local>

As you can see the lo interface is used.

There is no reason to disable this behavior. If you somehow manage to emit a packet to 192.168.0.1 through eth1 guess what: you won't receive it on the interface it was emitted from, so it will be lost. You could imagine further a whole convoluted setup, including configuring the switch port where eth1 is plugged to send back to its emitter traffic it received, but sooner or later this will defeat the original purpose of the experiment and the experiment won't work as intended anymore.

To do correct tests when dealing with networking one should always separate local tests and remote tests. If no remote system is available, this can be simulated by using an other network namespace which has its own separate complete network stack. Below is an example (that will not be affected by OP's tc ... netem).

As root user:

ip netns add remote
ip link add name vethtest up type veth peer netns remote name veth0
ip address add 192.0.2.1/24 dev vethtest
ip -n remote link set dev veth0 up
ip -n remote address add 192.0.2.2/24 dev veth0
ip netns exec remote ping 192.0.2.1

There are ways to reuse the original interface but this would require some disruption in configuration.

A.B
  • 31,762
  • 2
  • 62
  • 101
  • Thanks for the answer. That's very helpful. In my case, I have to run multiple processes on the same machine emulating different servers. Can the proposed commands extend to allow them connect together? Do I need to use bridge? Any pointer to relevant resources is appreciated. – renyuneyun Sep 21 '22 at 15:31
  • @renyuneyun In the end you could run containers (eg: LXC for system-oriented containers, Docker for application-oriented containers ...) for each separate server, instead of creating everything yourself like bridges and veth interfaces. – A.B Sep 21 '22 at 18:22
  • Thanks for the suggestion. Docker seems to be too heavy for my case, as I will need hundreds of instances/containers -- I'm running some experiment with decentralized context. I never used LXC directly, but will look into it. – renyuneyun Sep 22 '22 at 09:37