linux 2 interfaces 2 gateway routing problem

Question

I have a server, running Oracle Linux 8.4 with 2 IP addresses on 2 nics:

    link/ether 34:48:ed:f6:d3:5c brd ff:ff:ff:ff:ff:ff
    inet 10.154.224.252/24 brd 10.154.224.255 scope global noprefixroute eno1
       valid_lft forever preferred_lft forever
    inet6 fe80::3648:edff:fef6:d35c/64 scope link
       valid_lft forever preferred_lft forever
3: ens2f0np0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether bc:97:e1:7a:41:e0 brd ff:ff:ff:ff:ff:ff
    inet 10.154.226.252/24 brd 10.154.226.255 scope global noprefixroute ens2f0np0
       valid_lft forever preferred_lft forever
    inet6 fe80::be97:e1ff:fe7a:41e0/64 scope link
       valid_lft forever preferred_lft forever

So I created 2 table routing in /etc/iproute2/rt_tables db and app and script

#!/bin/sh
ip route add 10.154.226.0/24 dev ens2f0np0 src 10.154.226.252 table db
ip route add default via 10.154.226.1 dev ens2f0np0 table db
ip rule add from 10.154.226.252/24 table db
ip rule add to 10.154.226.252/24 table db

ip route add 10.154.224.0/24 dev eno1 src 10.154.224.252 table app
ip route add default via 10.154.224.1 dev eno1 table app
ip rule add from 10.154.224.252/24 table app
ip rule add to 10.154.224.252/24 table app 

ip r command show:
 
10.154.224.0/24 dev eno1 proto kernel scope link src 10.154.224.252 metric 100
10.154.226.0/24 dev ens2f0np0 proto kernel scope link src 10.154.226.252 metric 100
192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1 linkdown

i can connect to machine in any interface via ssh, but i cannot go out from server to internet, other subnets.

Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
10.154.224.0    0.0.0.0         255.255.255.0   U     100    0        0 eno1
10.154.226.0    0.0.0.0         255.255.255.0   U     100    0        0 ens2f0np0
192.168.122.0   0.0.0.0         255.255.255.0   U     0      0        0 virbr0

Something I miss? How can i fix that?

You can check what happens about Internet by using `ip route get ...`. Eg: `ip route get from 8.8.8.8 iif ens2f0np0 to 10.154.226.252` vs `ip route get from 10.154.226.252 to 8.8.8.8`: they must match. For other subnets, nothing is automatic: they must be replicated too — A.B, Jul 23 '21 at 17:56
@A.B `ip route get from 10.154.226.252 to 192.168.13.199 192.168.13.199 from 10.154.226.252 via 10.154.226.1 dev ens2f0np0 table db uid 0 cache` and ` ip route get from 10.154.224.252 to 192.168.13.199 192.168.13.199 from 10.154.224.252 via 10.154.224.1 dev eno1 table app uid 0 cache` all looks good, but not working :( what do u mean " For other subnets, nothing is automatic: they must be replicated too" ? it is internal network, so what should be replicated on localhost ? — kicek911, Jul 26 '21 at 11:12
can't tell you wrote only half of my example, so I can't tell if the reverse path matches. — A.B, Jul 26 '21 at 18:37
@A.B oh, sorry, this is a full answer: `ip route get from 8.8.8.8 iif ens2f0np0 to 10.154.226.252 local 10.154.226.252 from 8.8.8.8 dev lo table local` vs `ip route get from 10.154.226.252 to 8.8.8.8 8.8.8.8 from 10.154.226.252 via 10.154.226.1 dev ens2f0np0 table db uid 0 cache` so get from internet going to local table not to DB table.. — kicek911, Jul 27 '21 at 07:35
Everything looks coherent. Which means I don't have a clue on why it's not working. Not running anything fancy on the same system, like Docker or any network related application? — A.B, Jul 27 '21 at 08:39
@A.B no, this is a clean server, maybe i try it on other srv — kicek911, Jul 27 '21 at 11:44
Oh, for this to work you have to specifically bind the source IP address on the client socket. "i cannot go out from server to internet" <-- what's the precise command used to do the test? — A.B, Jul 27 '21 at 11:53
install socat and try `socat -d -d tcp4:192.168.13.199:80,bind=10.154.226.252 -` (replace 80 with the correct port). This one should work. — A.B, Jul 27 '21 at 11:56

score 2 · Answer 1 · answered Aug 12 '21 at 18:58

The reason it doesn't work to reach Internet is because the main routing table has no default route.

So for example when a TCP socket not bound (ie: bound to INADDR_ANY) is created and connects (using connect(2)), a route lookup is used to determine the local IP address to assign to the socket. This route uses a local address of INADDR_ANY because the socket wasn't bound to determine what will be the result. So it won't match the policy rules with a specific source address, since the value compared is 0.0.0.0 and won't match anything either in the main routing table.

This can be checked using ip route get:

# ip route get 8.8.8.8
RTNETLINK answers: Network is unreachable

but if the client application binds the address (example: curl --interface=10.154.226.252 ifconfig.co but as DNS is also affected, this might also be required to be added to this example: --dns-interface=10.154.226.252 if the DNS server isn't in the reachable LANs) then policy rules match and alternate routes apply, allowing the connection to happen:

# ip route get from 10.154.226.252 104.21.25.86
104.21.25.86 from 10.154.226.252 via 10.154.226.1 dev ens2f0np0 table db uid 0 
    cache 
# ip route get from 10.154.224.252 104.21.25.86
104.21.25.86 from 10.154.224.252 via 10.154.224.1 dev eno1 table app uid 0 
    cache

So a "default" default route is still needed in the main table if the client application won't explicitly bind to an IP address belonging to the host, and a choice must be made. Let's the app side is chosen:

ip route add default via 10.154.224.1

So now:

# ip route    
default via 10.154.224.1 dev eno1 
10.154.224.0/24 dev eno1 proto kernel scope link src 10.154.224.252 
10.154.226.0/24 dev ens2f0np0 proto kernel scope link src 10.154.226.252 
192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1 

# ip route get 8.8.8.8
8.8.8.8 via 10.154.224.1 dev eno1 src 10.154.224.252 uid 0 
    cache

If one really doesn't want to add a default route, it can be replaced with an ip rule command instead crafted in order to get the route lookup from INADDR_ANY to succeed:

ip route del default

ip rule add from 0.0.0.0/32 lookup app

Same result, except it's using the app table instead of the main table:

# ip route get 8.8.8.8
8.8.8.8 via 10.154.224.1 dev eno1 table app src 10.154.224.252 uid 0 
    cache

If one doesn't care which interface and would like to load-balance, instead of the default route above one can use a multipath route:

ip route del default # was already done
ip rule del from 0.0.0.0/32 lookup app

ip route add default nexthop via 10.154.224.1 dev eno1 nexthop via 10.154.226.1 dev ens2f0np0

The outgoing address will be chosen "randomly" (actually following some hash algorithm depending on source and destination, getting a stable route), selecting the hinted source address on the chosen interface to bind to, and once set any further outgoing packet in this connection will use the policy route (but even without policy route the hash would keep it on the same interface, it's not round-robin).

example (which might be different on an other system, try at least 4 sequential IP addresses):

# ip route get 8.8.8.9
8.8.8.9 via 10.154.224.1 dev eno1 src 10.154.224.252 uid 0 
    cache 
# ip route get 8.8.8.10
8.8.8.10 via 10.154.226.1 dev ens2f0np0 src 10.154.226.252 uid 0 
    cache

There are probably various caveats with multipath used as-is: proper SNAT/MASQUERADE is still required somewhere (but should work correctly), if a route fails with an interface, half the attempted client-without-binding connections will fail without any recovery mechanism, while before it would have been either 100% or 0%.

score 0 · Answer 2 · answered Jul 23 '21 at 15:30

0

You can not have two default gateways on a single Linux machine. Set the default gateway only for the interface via which you want to access the internet.

answered Jul 23 '21 at 15:30

Farhad Kia

78
6

So it is impossible to get separate network on 1 physical machine? When I set 1 default gateway and routing external FW see traffic with this interface .. – kicek911 Jul 23 '21 at 15:47
1

This is not true in multiple ways. Policy routing via `ip rule` (as in OP's post) is one of the mechanisms specifically meant to allow this. (SADR routing in IPv6 is another.) – u1686_grawity Jul 23 '21 at 16:30
@kicek911 may be this answer could help: [link](https://unix.stackexchange.com/questions/345862/is-it-possible-to-have-multiple-default-gateways-for-outbound-connections) – Farhad Kia Jul 27 '21 at 06:54
1

You cannot have two default gateways in a single routing table. You can have multiple routing tables though, each applied through different criteria – roaima Aug 12 '21 at 19:21

linux 2 interfaces 2 gateway routing problem

2 Answers2