Two NICs, two gateways, multiple aliases under each

Question

Last year I set up a CentOS 6 machine with two NICs, each using a separate gateway. Multiple aliases exist under NIC#1 (em1, em1:0, em1:1, ect), but currently only one interface exists under NIC#2 (em2).

I need to add an aliases to NIC#2 and am unsure how best to proceed.

Here is a quick summary of how I arrived at a working config as of now:

/etc/sysconfig/network-scripts/

em1
    DEVICE=em1
    NAME=siteABC
    IP=22.22.22.214
    GATEWAY=22.22.22.2
    NETMASK=255.255.255.0
    GATEWAY=22.22.22.2
    DNS1=1.1.1.1
    DNS2=1.1.1.2
    TYPE=Ethernet
    ONBOOT=yes
    BOOTPROTO=none

em1:0
    DEVICE=em1:0
    NAME=siteABC
    IPADDR=22.22.22.215
    GATEWAY=22.22.22.2
    NETMASK=255.255.255.0
    GATEWAY=22.22.22.2
    DNS1=1.1.1.1
    DNS2=1.1.1.2
    TYPE=Ethernet
    ONBOOT=yes
    BOOTPROTO=none

em2 
    DEVICE=em2
    NAME=sitetwo
    HWADDR=$MACADDRESS
    IPADDR=44.44.44.69
    NETMASK=255.255.255.0
    #GATEWAY=  ### commented out per below ###
    DNS1=1.1.1.1
    DNS2=1.1.1.2
    TYPE=Ethernet
    ONBOOT=yes
    BOOTPROTO=none

/etc/iproute2/rt_tables was edited to add a new line at the bottom (below #1 inr.ruhep). The new line contains 200 MGMT.

#
# reserved values
#
255 local
254 main
253 default
0 unspec
#
# local
#
#1 inr.ruhep
200 MGMT

/etc/sysconfig/network-scripts/route-em2 was created with the following:

default table MGMT via 44.44.44.2

** Note that 44.44.44.2 is the gateway of the second NIC (the public WAN). **

Lastly, I created a rules file called /etc/sysconfig/network-scripts/rule-em2 with the following:

from 44.44.44.69 table MGMT

Continued:

The default gateway has multiple aliases configured (em1, em1:0, em1:1, ect). The second gateway only has one interface (em2) however I need to add alias em2:0 for another site IP hosted by the machine.

What I've tried today

I configured an alias called em2:0 which mirrors em2 except for DEVICE, NAME, and IPADDR. The network restarted without a problem, but I'm not able to access the newly configured IPADDR under em2:0.

Question

Do I just need to add another entry in the /etc/sysconfig/network-scripts/rule-em2 file (shown above)? Or is there more to do to add an alias under em2? This is a production system and I am hesitant to make a quick change to see what happens.

You can only have one default gateway. You'll need to set route(s) to other networks specifically to use your 2nd interface. — ivanivan, Feb 16 '18 at 23:20
@A.B, thanks for the reminder on terminology. I've updated the question so not to confuse anyone — a coder, Feb 16 '18 at 23:23
I'll wait till I can be at the machine before adding the new rule. Will follow up then. — a coder, Feb 16 '18 at 23:24
you can have multiple default routes but one is picked at random for each packet (unless one of the routes has a lower, i.e. higher precedence, metric). see https://unix.stackexchange.com/questions/345862/is-it-possible-to-have-multiple-default-gateways-for-outbound-connections. but you shouldn't need to change any routing rules to add an IP address to an interface - unless the new IP is in a subnet that isn't already routed to your host or LAN segment (and even then, the new route should be on your router, not your host). — cas, Feb 17 '18 at 00:01
@A.B you already posted that link. how is it relevant to anything i said? adding a new IP within an already routed subnet to an interface does not require any routing changes. oh, and "random" is only semi-random - the source IP address of any outgoing packet affects the interface chosen (e.g. packets with src ip in 22.22.22.0/24 will go via em1, packets from 44.44.44.0/24 will go via em2). I glossed over that because it's not particularly relevant here - I was mostly just pointing out that the comments saying you could only have one default gateway were incorrect. — cas, Feb 17 '18 at 00:18

A.B · Accepted Answer · 2018-02-17T02:13:07.573

Yes it appears you have to add a rule for the new IP 44.44.44.X on em2. That would be ip rule add from 44.44.44.X table MGMT.

Without rule to the MGMT table (200), the default route chosen would be default via 22.22.22.2 dev em1, even for this IP on em2. With this setting a customer from Internet trying to reach this new management IP would have her connection arriving at em2 from an IP with no default route (so no route) defined via em2: the default setting for CentOS' reverse path filtering would drop the packet.

Likewise, an outgoing packet with this new IP 44.44.44.X would attempt to go to the customer via the only available default route: 22.22.22.2 on em1. It's possible (but not very relevant, the other way is already not working) that an other equipment in the path would not allow this either (eg: if 22.22.22.2 and 44.44.44.2 are going via two different Internet providers, or simply if 22.22.22.2 has no Internet connectivity).

Adding ip rule add from 44.44.44.X table MGMT will select this routing table when the source ip is 44.44.44.X, now selecting default via 44.44.44.2 dev em2 from table MGMT. A packet from Internet arriving at 44.44.44.X via em2 has now its route through em2, so won't be dropped by rp_filter. An outgoing packet from 44.44.44.X will now correctly use em2 to go to Internet via 44.44.44.2.

You can easily see the difference by typing for example:

ip route get 8.8.8.8 from 44.44.44.X

Without the rule, you'll see the route 8.8.8.8 from 44.44.44.X via 22.22.22.2 dev em1, with it it will switch to 8.8.8.8 from 44.44.44.X via 44.44.44.2 dev em2 table 200

Two NICs, two gateways, multiple aliases under each

1 Answers1