I'm trying to set a per-ip connection limit on a kubernetes node using iptables connlimit. Since each container on the VM has a different source IP (overlay net), using connlimit should work. I added the rule
iptables -I FORWARD 1 -p tcp --syn -m connlimit --connlimit-above 25 --connlimit-mask 32 -j REJECT --reject-with tcp-reset
I'm testing by having a container make a bunch of connections in burst of 50 to an external service. This initially works in that first burst. 25 connect and 25 have connection refused. I can see 25 entries in conntrack table with the correct source ip.
When I wait a few seconds and run the script to try to create another 50 connections (while 25 are still in ESTABLISHED), it allows another 25 through. I can now see 50 connections with that same source IP in the conntrack table, all ESTABLISHED. The desired result is that none will go through as long as the original 25 are still connected.
What am I doing wrong?
I'm using forward instead of input chain because of how container networking is configured.