Introduction
I have following load balancing configuration:
10.0.1.31 - lb
10.0.1.35 - virtual IP
10.0.1.32 - node1 (tomcat + mysql)
10.0.1.33 - node2 (tomcat + mysql)
I'm using keepalived which is redirecting packets to active nodes - shared ip address is 10.0.1.35. lb has required config:
echo "net.ipv4.ip_nonlocal_bind = 1" >> /etc/sysctl.conf
node1 and node2 to handle properly incoming packets needs to have 10.0.1.35 on lo
[root@lb-node1 ~]# ip addr list dev lo
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet 10.0.1.35/32 scope global lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
Problem
Due to the fact that we have 10.0.1.35 on the lo if the locale tomcat is trying to connect to mysql via lb - connection is established to local instance.
I would like skip lo for outgoing packets
Test
From node1 which has disabled mysql service and 10.0.1.35 ip on lo interface I'm trying to connect to mysql on the node2. Unfortunately the result is
[root@lb-node1 ~]# telnet 10.0.1.35 3306
Trying 10.0.1.35...
telnet: connect to address 10.0.1.35: Connection refused
of course if I remove 10.0.1.35 IP from lo interface I'm able to connect to mysql instance on node2
Solution(?)
I was trying do add routes with appropriate metrics but it doesn't help :/
[root@lb-node1 ~]# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
10.0.1.0 0.0.0.0 255.255.255.255 UH 0 0 0 eth0
10.0.1.0 0.0.0.0 255.255.255.255 UH 100 0 0 lo
169.254.0.0 0.0.0.0 255.255.0.0 U 1002 0 0 eth0
0.0.0.0 10.0.1.1 0.0.0.0 UG 0 0 0 eth0
@Patrick Solution
vip=10.0.1.35
ip route add local $vip dev lo table 10 proto kernel scope host
ip rule add to $vip lookup 10 prio 1
ip route del local $vip dev lo table local
ip rule add to $vip iif lo lookup main prio 0
@Patrick Solution Issue
Initial state
VIP + your ip route configuration, mysql is working on both nodes
P Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 10.0.1.35:http rr persistent 6
TCP 10.0.1.35:mysql rr persistent 6
-> 10.0.1.32:mysql Route 10 0 0
-> 10.0.1.33:mysql Route 10 0 2
UDP 10.0.1.35:snmptrap rr persistent 6
-> 10.0.1.32:snmptrap Route 10 0 0
-> 10.0.1.33:snmptrap Route 10 0 1
root@lb-node1 ~]# mysql -h 10.0.1.35 -u test -p
Enter password:
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MariaDB connection id is 126
Server version: 5.5.36-MariaDB-wsrep-log MariaDB Server, wsrep_25.9.r3961
Copyright (c) 2000, 2014, Oracle, Monty Program Ab and others.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
MariaDB [(none)]> SHOW VARIABLES WHERE Variable_name = 'hostname';
+---------------+----------+
| Variable_name | Value |
+---------------+----------+
| hostname | lb-node2 |
+---------------+----------+
1 row in set (0.00 sec)
MariaDB [(none)]>
[root@lb-node2 ~]# mysql -h 10.0.1.35 -u test -p
Enter password:
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MariaDB connection id is 133
Server version: 5.5.36-MariaDB-wsrep-log MariaDB Server, wsrep_25.9.r3961
Copyright (c) 2000, 2014, Oracle, Monty Program Ab and others.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
ariaDB [(none)]> ;
ERROR: No query specified
MariaDB [(none)]> SHOW VARIABLES WHERE Variable_name = 'hostname';
+---------------+----------+
| Variable_name | Value |
+---------------+----------+
| hostname | lb-node2 |
+---------------+----------+
1 row in set (0.00 sec)
MariaDB [(none)]>
As you can see everything is working correctly.
Issue
But when I shut down currently active mysql server:
[root@lb-node2 ~]# service mysql stop
Shutting down MySQL.... SUCCESS!
[root@lb-node2 ~]#
Every 2.0s: ipvsadm -l Fri May 9 10:20:49 2014
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 10.0.1.35:http rr persistent 6
TCP 10.0.1.35:mysql rr persistent 6
-> 10.0.1.32:mysql Route 10 0 0
UDP 10.0.1.35:snmptrap rr persistent 6
-> 10.0.1.32:snmptrap Route 10 0 0
-> 10.0.1.33:snmptrap Route 10 0 1
I cannot connect to mysql from both nodes
[root@lb-node2 ~]# mysql -h 10.0.1.35 -u test -p
Enter password:
.....
It seems to me that node1 is not accepting incoming packets, because load balancer is correctly redirecting packets
[root@lb-node1 ~]# tcpdump -i eth0 'port 3306' and src 10.0.1.33 or dst 10.0.1.33
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
10:35:26.915640 ARP, Reply 10.0.1.35 is-at 52:54:00:30:a3:4b (oui Unknown), length 28
10:35:26.915987 IP 10.0.1.33.38517 > 10.0.1.35.mysql: Flags [S], seq 2024730796, win 14600, options [mss 1460,sackOK,TS val 1298907 ecr 0,nop,wscale 7], length 0
10:35:27.914788 IP 10.0.1.33.38517 > 10.0.1.35.mysql: Flags [S], seq 2024730796, win 14600, options [mss 1460,sackOK,TS val 1299907 ecr 0,nop,wscale 7], length 0
10:35:29.914784 IP 10.0.1.33.38517 > 10.0.1.35.mysql: Flags [S], seq 2024730796, win 14600, options [mss 1460,sackOK,TS val 1301907 ecr 0,nop,wscale 7], length 0
What is strange because still I have VIP on lo
[root@lb-node1 ~]# ip addr list dev lo
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet 10.0.1.35/32 scope global lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
[root@lb-node1 ~]# ip route
10.0.1.0/24 dev eth0 proto kernel scope link src 10.0.1.32
169.254.0.0/16 dev eth0 scope link metric 1002
default via 10.0.1.1 dev eth0
