I have above scenario where server has bond0 interface connected to SW1 and SW2 relevant interface of server, last night somehow SW1 got rebooted and server lost eth0 connectivity but it didn't failover to eth1 also every after switch rebooted successfully network was down and this happened on multiple server not single. so finally i have restart network on all server to make it work. we have primary=eth0 set in bond0 configuration.
ifcfg-bond0
DEVICE=bond0
BOOTPROTO=none
ONBOOT=yes
BONDING_OPTS=mode=1 arp_interval=1000 arp_ip_target=192.168.10.1 miimon=500 downdelay=1000 primary=eth0 primary_reselect=always
NETMASK=255.255.255.0
IPADDR=192.168.10.20
GATEWAY=192.168.10.1
ifcfg-eth0
DEVICE=eth0
BOOTPROTO=none
ONBOOT=yes
MASTER=bond0
SLAVE=yes
ifcfg-eth1
DEVICE=eth1
BOOTPROTO=none
ONBOOT=yes
MASTER=bond0
SLAVE=yes
This is what i am seeing in logs when this happened
[root@server1 ~]# cat /var/log/messages | grep eth
Jun 8 05:03:05 [email protected] kernel: : bnx2 0000:02:00.0: eth0: NIC Copper Link is Down
Jun 8 05:03:05 [email protected] kernel: : bonding: bond0: link status definitely down for interface eth0, disabling it
Jun 8 05:03:05 [email protected] kernel: : bonding: bond0: making interface eth1 the new active one.
Jun 8 05:07:03 [email protected] kernel: : bnx2 0000:02:00.0: eth0: NIC Copper Link is Up, 1000 Mbps full duplex
Jun 8 05:07:03 [email protected] kernel: : bonding: bond0: link status definitely up for interface eth0.
Jun 8 05:20:45 [email protected] kernel: : bonding: bond0: Removing slave eth0
Jun 8 05:20:45 [email protected] kernel: : bonding: bond0: Warning: the permanent HWaddr of eth0 - 18:A9:05:3A:39:1D - is still in use by bond0. Set the HWaddr of eth0 to a different address to avoid conflicts.
Jun 8 05:20:45 [email protected] kernel: : bonding: bond0: releasing backup interface eth0
Jun 8 05:20:46 [email protected] kernel: : bonding: bond0: Removing slave eth1
Jun 8 05:20:46 [email protected] kernel: : bonding: bond0: releasing active interface eth1
Jun 8 05:20:46 [email protected] kernel: : bonding: bond0: Adding slave eth0.
Jun 8 05:20:46 [email protected] kernel: : bnx2 0000:02:00.0: eth0: using MSIX
Jun 8 05:20:46 [email protected] kernel: : bonding: bond0: enslaving eth0 as a backup interface with a down link.
Jun 8 05:20:46 [email protected] kernel: : bonding: bond0: Adding slave eth1.
Jun 8 05:20:46 [email protected] kernel: : bnx2 0000:02:00.1: eth1: using MSIX
Jun 8 05:20:46 [email protected] kernel: : bonding: bond0: enslaving eth1 as a backup interface with a down link.
Jun 8 05:20:49 [email protected] kernel: : bnx2 0000:02:00.0: eth0: NIC Copper Link is Up, 1000 Mbps full duplex
Jun 8 05:20:49 [email protected] kernel: : bonding: bond0: link status definitely up for interface eth0.
Jun 8 05:20:49 [email protected] kernel: : bonding: bond0: making interface eth0 the new active one.
Jun 8 05:20:49 [email protected] kernel: : bnx2 0000:02:00.1: eth1: NIC Copper Link is Up, 1000 Mbps full duplex
Jun 8 05:20:50 [email protected] kernel: : bonding: bond0: link status definitely up for interface eth1.
