VLAN switching between trunk ports on CentOS

Question

I am using CentOS Linux release 8.3.2011 and am struggling to set it up as a switch. The network topology is as follows:

[switch1]--------[vsphere]--------------[centos]--------------------[vRouter]
     gi0/0     vmnic2   SW1-trunk   end224   lnkbe296edcl2b     Gi0/0.10
                     SW1                bridge11

Topology 1

So, in summary:

A physical switch, switch1, with interface gi0/0 configured as trunk and with some switched virtual interfaces configured with IP addresses for testing. For this test, I will use the interface vlan 10, which has the IP address 192.168.0.10/24;
vSphere hypervisor, with a vSwitch SW1, containing port group "SW1 - trunk" and a physical adapter vmnic2. The port group is configured with VLAN ID 4095, which is the way to configure a port as trunk on vmware;
A CentOS VM, called CML, which uses KVM to host virtual network functions. I've configured a virtual router on top of it. I've created bridge11 and associated the vmware facing interface (ens224) and vrouter facing interface (lnkbe296edcl2b) to it.
A vRouter, which is an IOSv (virtualized Cisco IOS), with a IP subinterface with IP address 192.168.0.11/24.

vSphere topology below:

Well, with this configuration, I cannot ping from vRouter (192.168.0.11) to switch1 SVI (192.168.0.10).

R1#sh ip int br
Interface                  IP-Address      OK? Method Status                Protocol
GigabitEthernet0/0         unassigned      YES manual up                    up      
GigabitEthernet0/0.10      192.168.0.11    YES TFTP   up                    up      
GigabitEthernet0/1         unassigned      YES unset  administratively down down    
GigabitEthernet0/2         unassigned      YES unset  administratively down down    
GigabitEthernet0/3         unassigned      YES unset  administratively down down    
R1#ping 192.168.0.10
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.0.10, timeout is 2 seconds:
.....
Success rate is 0 percent (0/5)
R1#sh arp
Protocol  Address          Age (min)  Hardware Addr   Type   Interface
Internet  192.168.0.10            0   Incomplete      ARPA   
Internet  192.168.0.11            -   5254.0000.b61d  ARPA   GigabitEthernet0/0.10
R1#

If I change the network topology to the one below:

[switch1]--------[vsphere]--------------[centos]--------------------[vRouter]
     gi0/0     vmnic2   SW1-trunk  end224.10   lnkbe296edcl2b     Gi0/0
                     SW1                bridge11

Topology 2 In this topology:

I am using a subinterface on centos towards vmware, so, it receives untagged traffic from the bridge and send it tagged to VMWare port group SW1-trunk;
I am using the main interface in the router. So, the traffic leaves untagged;
I've added an IP address to bridge11 - 192.168.0.1/24, for testing.

In this scenario, I can ping from bridge11 to switch1 and from bridge11 to vRouter, but not from vRouter to switch1.

CentOS tests:

[~]$ping -c5 -q 192.168.0.11
PING 192.168.0.11 (192.168.0.11) 56(84) bytes of data.

--- 192.168.0.11 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 11ms
rtt min/avg/max/mdev = 1.718/2.151/2.879/0.407 ms
[~]$ping -c5 -q 192.168.0.10
PING 192.168.0.10 (192.168.0.10) 56(84) bytes of data.

--- 192.168.0.10 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 103ms
rtt min/avg/max/mdev = 0.398/0.515/0.656/0.090 ms
[~]$ip nei show dev bridge11
192.168.0.10 lladdr 00:12:43:7e:4b:c1 STALE
192.168.0.11 lladdr 52:54:00:00:b6:1d STALE
[~]$

vRouter (R1) tests:

R1#ping 192.168.0.1 
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.0.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/2/4 ms
R1#ping 192.168.0.10
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.0.10, timeout is 2 seconds:
.....
Success rate is 0 percent (0/5)
R1#show arp         
Protocol  Address          Age (min)  Hardware Addr   Type   Interface
Internet  192.168.0.1             0   000c.2997.7368  ARPA   GigabitEthernet0/0
Internet  192.168.0.10            0   Incomplete      ARPA   
Internet  192.168.0.11            -   5254.0000.b61d  ARPA   GigabitEthernet0/0
R1#

I've searched and only found references with linux untagging traffic and routing... I need linux to behave as a vlan switch without need to look at the ip headers.

Any hints on how to make CentOS to switch vlan traffic, from trunk to trunk port?

Thanks in advance

edit (5/16/21) - to include outputs from commands asked by A.B.:

[sysadmin]$sysctl -ne net.bridge.bridge-nf-call-iptables
0
[sysadmin]$tcpdump -e -n -i lnk2fb2a645ld1
dropped privs to tcpdump
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on lnk2fb2a645ld1, link-type EN10MB (Ethernet), capture size 262144 bytes
18:00:26.276792 52:54:00:00:b6:1d > 01:00:0c:cc:cc:cc, 802.3, length 396: LLC, dsap SNAP (0xaa) Individual, ssap SNAP (0xaa) Command, ctrl 0x03: oui Cisco (0x00000c), pid CDP (0x2000), length 388: CDPv2, ttl: 180s, Device-ID 'R1', length 388
18:00:31.178085 52:54:00:00:b6:1d > 52:54:00:00:b6:1d, ethertype Loopback (0x9000), length 60: Loopback, skipCount 0, Reply, receipt number 0, data (40 octets)
18:00:35.482297 52:54:00:00:b6:1d > Broadcast, ethertype 802.1Q (0x8100), length 64: vlan 10, p 0, ethertype ARP, Request who-has 192.168.0.10 tell 192.168.0.11, length 46
18:00:39.480774 52:54:00:00:b6:1d > Broadcast, ethertype 802.1Q (0x8100), length 64: vlan 10, p 0, ethertype ARP, Request who-has 192.168.0.10 tell 192.168.0.11, length 46
18:00:41.177503 52:54:00:00:b6:1d > 52:54:00:00:b6:1d, ethertype Loopback (0x9000), length 60: Loopback, skipCount 0, Reply, receipt number 0, data (40 octets)
18:00:41.480474 52:54:00:00:b6:1d > Broadcast, ethertype 802.1Q (0x8100), length 64: vlan 10, p 0, ethertype ARP, Request who-has 192.168.0.10 tell 192.168.0.11, length 46
18:00:41.655353 52:54:00:00:b6:1d > ab:00:00:02:00:00, ethertype MOP RC (0x6002), length 77: 
        0x0000:  3d00 0700 0000 0100 0303 0000 0200 0221  =..............!
        0x0010:  0003 0006 0000 0000 0000 0400 023c 0005  .............<..
        0x0020:  0002 d805 0600 0200 0107 0006 5254 0000  ............RT..
        0x0030:  b61d 6400 0179 9001 0101 9101 02ee 05    ..d..y.........
18:00:43.480161 52:54:00:00:b6:1d > Broadcast, ethertype 802.1Q (0x8100), length 64: vlan 10, p 0, ethertype ARP, Request who-has 192.168.0.10 tell 192.168.0.11, length 46
^C
8 packets captured
8 packets received by filter
0 packets dropped by kernel
[sysadmin]$

edit (5/16/21) - to include additional command outputs requested by A.B.:

$ip link show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: ens160: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master bridge0 state UP mode DEFAULT group default qlen 1000
    link/ether 00:0c:29:97:73:54 brd ff:ff:ff:ff:ff:ff
3: ens192: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master bridge1 state UP mode DEFAULT group default qlen 1000
    link/ether 00:0c:29:97:73:5e brd ff:ff:ff:ff:ff:ff
4: ens224: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master bridge11 state UP mode DEFAULT group default qlen 1000
    link/ether 00:0c:29:97:73:68 brd ff:ff:ff:ff:ff:ff
5: ens256: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 00:0c:29:97:73:72 brd ff:ff:ff:ff:ff:ff
6: bridge0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether 00:0c:29:97:73:54 brd ff:ff:ff:ff:ff:ff
7: bridge1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether 00:0c:29:97:73:5e brd ff:ff:ff:ff:ff:ff
10: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default qlen 1000
    link/ether 52:54:00:90:2d:88 brd ff:ff:ff:ff:ff:ff
11: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc fq_codel master virbr0 state DOWN mode DEFAULT group default qlen 1000
    link/ether 52:54:00:90:2d:88 brd ff:ff:ff:ff:ff:ff
44: bridge11: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether 00:0c:29:97:73:68 brd ff:ff:ff:ff:ff:ff
45: lnk2fb2a645ld1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master bridge11 state UNKNOWN mode DEFAULT group default qlen 1000
    link/ether f2:ca:f4:6e:96:14 brd ff:ff:ff:ff:ff:ff
$ip -br link show
lo               UNKNOWN        00:00:00:00:00:00 <LOOPBACK,UP,LOWER_UP> 
ens160           UP             00:0c:29:97:73:54 <BROADCAST,MULTICAST,UP,LOWER_UP> 
ens192           UP             00:0c:29:97:73:5e <BROADCAST,MULTICAST,UP,LOWER_UP> 
ens224           UP             00:0c:29:97:73:68 <BROADCAST,MULTICAST,UP,LOWER_UP> 
ens256           UP             00:0c:29:97:73:72 <BROADCAST,MULTICAST,UP,LOWER_UP> 
bridge0          UP             00:0c:29:97:73:54 <BROADCAST,MULTICAST,UP,LOWER_UP> 
bridge1          UP             00:0c:29:97:73:5e <BROADCAST,MULTICAST,UP,LOWER_UP> 
virbr0           DOWN           52:54:00:90:2d:88 <NO-CARRIER,BROADCAST,MULTICAST,UP> 
virbr0-nic       DOWN           52:54:00:90:2d:88 <BROADCAST,MULTICAST> 
bridge11         UP             00:0c:29:97:73:68 <BROADCAST,MULTICAST,UP,LOWER_UP> 
lnk2fb2a645ld1   UNKNOWN        f2:ca:f4:6e:96:14 <BROADCAST,MULTICAST,UP,LOWER_UP> 
$ip -br address
lo               UNKNOWN        127.0.0.1/8 ::1/128 
ens160           UP             
ens192           UP             
ens224           UP             
ens256           UP             
bridge0          UP             10.30.8.241/22 fe80::20c:29ff:fe97:7354/64 
bridge1          UP             x.x.x.x/25 fe80::20c:29ff:fe97:735e/64 
virbr0           DOWN           192.168.255.1/24 
virbr0-nic       DOWN           
bridge11         UP             fe80::20c:29ff:fe97:7368/64 
lnk2fb2a645ld1   UNKNOWN        
$

Can you verify that `sysctl -ne net.bridge.bridge-nf-call-iptables` displays either nothing or `0` but not `1` ? If it returns `1` does toggling it to 0 ( `sysctl -w net.bridge.bridge-nf-call-iptables=0`) change the behavior? — A.B, May 16 '21 at 19:53
Hi, thank you for the quick reply. The traffic is tagged, as expected, and the net.bridge.bridge-nf-call-iptables=0 ``` [sysadmin]$sysctl -ne net.bridge.bridge-nf-call-iptables 0 ``` — Andre, May 16 '21 at 21:01
I've edited the original text to include the commands outputs. The link name changed to lnk2fb2a645ld1 as it changes when I stop and start simulations in the VM who hosts the vRouter. — Andre, May 16 '21 at 21:08
anyway looks like for experiment 2 the lnk interface is not configured like end224 (with a vlan sub interface) while everything else uses a vlan. Make it symmetrical with the other and it will work. You could as well set the bridge vlan aware, but I suspect for both things, the issue is not how to do it manually but how to configure libvirt. I don't think I'd give any answer, everything is a bit too fuzzy. — A.B, May 16 '21 at 21:49
The output of the commands I sent was using the 1st configuration described in my original text. I am editing the text with the output at the end. — Andre, May 16 '21 at 22:01
My end goal was to make the scenario described in topology 1 to work. I've set the scenario in topology 2 just to make sure the vRouter and switch1 were accessible by CentOS. — Andre, May 16 '21 at 22:15
The question is not adapted to a minimal problem so it's a bit difficult to answer it. Anyway, please check these Q/A where I made answers: https://unix.stackexchange.com/questions/546136/bridged-interfaces-and-vlan-tags/546201#546201 , https://unix.stackexchange.com/questions/556735/linux-vlan-aware-bridges-and-trunk-ports/556765#556765 . Of course being a different problem, the use is different (here tagged both sides). Note that libvirt is probably not compatible with bridge vlan_filtering (it might be compatible with the openvswitch equivalent, but I don't have much knowledge there). — A.B, May 17 '21 at 17:38
"not compatible" just means manual actions have to be done after starting the VM. — A.B, May 17 '21 at 17:44

score 0 · Accepted Answer · answered May 17 '21 at 20:39

0

To make it work, I needed to do two things:

Disable vlan_filtering (ip link set dev bridge11 type bridge vlan_filtering 0)
Enable promiscuous mode and forged transmits on vmware vSphere (see below)

answered May 17 '21 at 20:39

Andre

1
2

vlan_filtering 0 is the default. This means your problem was caused only by VMware and is not an Unix/Linux problem. There was no possibility to solve it except by yourself. – A.B May 18 '21 at 19:57
Do you know how to check the vlan_filtering setting? I couldn't find a way to do so... – Andre May 20 '21 at 18:52
It was not a unix/linux problem, but I appreciate your quick response and guidance A.B. Using tcpdump to make sure the traffic was being forwarded was critical to determine the problem wasn't in Linux. It is important to know where the problem is, but equally important where it is not. – Andre May 20 '21 at 18:55

VLAN switching between trunk ports on CentOS

1 Answers1