nf_conntrack_sip does not work SOMETIMES, restarting iptables USUALLY fixes it

Question

I'm trying to use nf_conntrack_sip on box that is running Asterisk, that is, not routing traffic for another VoIP box. Setup works until I reboot. After reboot nf_conntrack_sip ALMOST always stops working and media traffic is dropped.

conntrack --dump | grep -E 'sip|helper'
# No output matching 'sip' nor 'helper' while a call is in progress (albeit no audio)

The iptables rules are loaded correctly confirmed by iptables-save.

Then I do systemctl restart iptables and 9/10 times that fixes it. If it does not then I restart repeat the iptables restart.

conntrack --dump | grep -E 'sip|helper'
conntrack v1.4.4 (conntrack-tools): 9 flow entries have been shown.
udp      17 3597 src=10.7.0.38 dst=10.47.1.11 sport=5063 dport=5060 src=10.47.1.11 dst=10.7.0.38 sport=5060 dport=5063 [ASSURED] mark=0 secctx=system_u:object_r:unlabeled_t:s0 helper=sip use=2

Simply reloading the rules with iptables-restore < /etc/sysconfig/iptables does not help. I suspect unloading/loading conntrack or some modules does the trick.

Occasionally it does work at boot, but very rare. Asterisk start quickly. Giving it more time to "finish starting something" does not help.

Update: restarting iptables while nf_conntrack_sip is working as expected, can, rarely, break it.

The setup:

Update: Initially the problem was described as occurring on a VM, but since then I reinstalled onto real hardware (i5-6500 CPU @ 3.20GHz with 8Gb RAM) with exactly the same problem still occurring. All identical packages (same provision script) as the initial VM.

The OS is CentOS-7.4 Minimal + updates, kernel 3.10.0-693.21.1.el7.x86_64. It is all installed from RPMs, no custom kernels nor modules. Update: I also did yum update to latest stable packages and kernel available from CentOS at 2018-08-10. The problem persists.

I did yum autoremove firewalld and yum install iptables-services.

Diffs to /etc/sysconfig/iptables-config (other values are defaults by RPM)

-IPTABLES_MODULES=""
+IPTABLES_MODULES="nf_conntrack_sip"

Added file /etc/modprobe.d/nf_conntrack.conf:

options nf_conntrack nf_conntrack_helper=0

The entire /etc/sysconfig/iptables is very simple:

*raw
-A PREROUTING -p udp --dport 5060 -j CT --helper sip
COMMIT

*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT
-A INPUT -p udp -m state --state NEW -m udp --dport 5060 -j ACCEPT
-A INPUT -j LOG --log-level 7 --log-prefix "REJECT in filter.INPUT:"
-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A FORWARD -j REJECT --reject-with icmp-host-prohibited
COMMIT

Update: Setting module options options nf_conntrack nf_conntrack_helper=1 and NOT using the iptables rule ... -j CT --helper sip does NOT fix it and the behavior remains non-deterministic.

Not relevant to the problem, only to confirm that packets are dropped, as opposed to having NAT issues, /etc/rsyslog.d/kern-debug.conf

kern.=debug /var/log/kernel-debug

Testing with a Cisco SPA504G phone that dials into the PBX and gets hold music. Not trying to do anything complicated with media. SIP signalling and Media are exchanged with same IPv4 address. The test call is only between the phone and the PBX. No other parties involved.

My attempt to diagnose it:

I've made short script that tries to capture the state of various things before and after the fix by restarting iptables, to compare by diff. The script:

for f in $( find /proc/sys/net/netfilter -type f ); do
  echo f=${f}
  cat "${f}"
done

echo cat /sys/module/nf_conntrack/parameters/*
cat /sys/module/nf_conntrack/parameters/*

echo ls /sys/module/nf_conntrack/holders/
ls /sys/module/nf_conntrack/holders/

echo cat /sys/module/nf_conntrack_sip/parameters/*
cat /sys/module/nf_conntrack_sip/parameters/*
echo ls /sys/module/nf_conntrack_sip/holders/
ls /sys/module/nf_conntrack_sip/holders/

echo ls /sys/module/ip*/holders/
ls /sys/module/{ip,nf_}*/holders/

echo sysctl -a
sysctl -a

echo lsmod
lsmod

echo iptables-save
iptables-save

The only thing I notice is that OFTEN module nf_conntrack_netlink IS listed as loaded after the boot, while there is a problem. Sometimes it is NOT listed by lsmod AFTER the boot but there is still the problem. After restarting iptables it is, to the best of my knowledge, never listed as loaded. I suspect it is unrelated because there is no direct link between it being loaded and the problem manifesting.

Answer 1

Solution

The solution was to simply mark the outgoing packets to be handled by conntrack sip helper too:

iptables -t raw -A OUTPUT -p udp -m udp --sport 5060 -j CT --helper sip

Cause

The problem was the firewall rule was marking only incoming packets for conntrack sip helper.

iptables -t raw -A PREROUTING -p udp -m udp --dport 5060 -j CT --helper sip

When the PBX was the one to send the first packet toward the phone, it would establish a conntrack entry without sip helper. The entry continued to match the SIP conversation without SIP helper being involved.

[root@test ~]# conntrack -L | grep -E '5060|sip'
conntrack v1.4.4 (conntrack-tools): 13 flow entries have been shown.
udp      17 159 src=10.47.1.11 dst=10.7.0.38 sport=5060 dport=1024 src=10.7.0.38 dst=10.47.1.11 sport=1024 dport=5060 [ASSURED] mark=0 secctx=system_u:object_r:unlabeled_t:s0 use=1

When the phone was the one to send the first packet toward the PBX, it would hit the rule with "-j CT --helper sip" listed above and the conntrack entry would get created with sip helper.

[root@test ~]# conntrack -L | grep -E '5060|sip'
conntrack v1.4.4 (conntrack-tools): 9 flow entries have been shown.
udp      17 3588 src=10.7.0.38 dst=10.47.1.11 sport=1024 dport=5060 src=10.47.1.11 dst=10.7.0.38 sport=5060 dport=1024 [ASSURED] mark=0 secctx=system_u:object_r:unlabeled_t:s0 helper=sip use=2

Note the "helper=sip" towards the end of the entry, compare with lack of it in first sample.

The PBX and Phone send SIP packets at each other to confirm the other is there, so the timing made it appear non-deterministic.

Asterisk preserves the state of peers across reboots, and probes them after reboot, thus being far more likely to send outgoing packet first and causing the non-SIP-helper entry in the conntrack.

Big thank you to user A.B for pointing me in the right direction in the comments.

Remaining mystery

What I can not explain is why when I had modprobe option

options nf_conntrack nf_conntrack_helper=0

It was still getting broken and "fixed" in the same way. I did not spend much time on helper automatically triggering, so maybe I did something wrong. I might update this answer if I find out more. I am not planning to use automatic helper enabled.