So the university IT security team and I have been going around and around on this with no breaks... anyone have any thoughts on this:
I recently set up a small file server for my lab running Debian 8.6 on a dedicated computer (Intel Avoton C2550 processor -- happy to provide more hardware info if needed, but I think unnecessary). Debian installed without any problems, and at the time I also installed Samba, NTP, ZFS, and python. Things seemed to be working fine, so I let it sit and run in the corner of the lab for a few weeks.
About two weeks ago, I received an email from the IT team saying there that my server has been "compromised" and is vulnerable being used in an NTP amplification/DDoS attack (NTP Amplification Attacks Using CVE-2013-5211 as described in https://www.us-cert.gov/ncas/alerts/TA14-013A ). The sign they pointed to was a lot of NTPv2 traffic on port 123. Weirdly, the IP address that they identified this coming from (*.*.*.233) was different from the ip address my server was configured for and reported via ifconfig (*.*.*.77). Nevertheless, some basic troubleshooting revealed that my computer was indeed generating this traffic on port 123 (as revealed by tcpdump).
Here is where the bizarreness began. I first ran through the "fixes" recommended for CVE-2013-5211 (both updating the NTP past version 4.2.7 as well as disabling monlist functionality). Neither stemmed the traffic flow. I then tried blocking the UDP 123 port via IP tables:
$ /sbin/iptables -A INPUT -o eth0 -p udp --destination-port 123 -j DROP
$ /sbin/iptables -A OUTPUT -o eth0 -p udp --destination-port 123 -j DROP
but that too had no effect on the traffic. I finally tried purging NTP from the system, but that had no effect on the traffic either. As of this afternoon, nmap was still reporting:
Starting Nmap 5.51 ( http://nmap.org ) at 2016-12-19 16:15 EST
Nmap scan report for *.233
Host is up (0.0010s latency).
PORT STATE SERVICE
123/udp open ntp
| ntp-monlist:
| Public Servers (2)
| 50.116.52.97 132.163.4.101
| Public Clients (39)
| 54.90.159.15 185.35.62.119 185.35.62.233 185.35.63.86
| 54.197.89.98 185.35.62.142 185.35.62.250 185.35.63.108
| 128.197.24.176 185.35.62.144 185.35.62.251 185.35.63.128
| 180.97.106.37 185.35.62.152 185.35.63.15 185.35.63.145
| 185.35.62.27 185.35.62.159 185.35.63.27 185.35.63.146
| 185.35.62.52 185.35.62.176 185.35.63.30 185.35.63.167
| 185.35.62.65 185.35.62.186 185.35.63.34 185.35.63.180
| 185.35.62.97 185.35.62.194 185.35.63.38 185.35.63.183
| 185.35.62.106 185.35.62.209 185.35.63.39 185.35.63.185
|_ 185.35.62.117 185.35.62.212 185.35.63.43
Which is all very weird since NTP has been purged from the system for weeks now.
After hitting a dead end on this path, I started thinking about the whole IP-address mismatch issue. My computer seemed to be sitting on both *.233 and *.77 IPs (as confirmed by successfully pinging both with ethernet cable attached and both unavailable with cable unplugged), but *.233 never shows up in ifconfig:
Link encap:Ethernet HWaddr d0:XX:XX:51:78:XX
inet addr:*.77 Bcast:*.255 Mask:255.255.255.0
inet6 addr: X::X:X:X:787a/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:23023571 errors:0 dropped:1362 overruns:0 frame:0
TX packets:364849 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:7441732389 (6.9 GiB) TX bytes:44699444 (42.6 MiB)
Memory:df300000-df37ffff
There is no reference to *.233 in /etc/network/interfaces, so I don't see where this IP assignment is coming from.
So, I have two likely related questions I'm hoping someone can help me with: 1) How can I eliminate this NTP traffic from spewing from my server to get IT off my back? 2) What is up with this second IP address my server is sitting on and how can I remove it?
Thanks, folks :)
UPDATE:
As requested:
$iptables -L -v -n
Chain INPUT (policy ACCEPT 57 packets, 6540 bytes)
pkts bytes target prot opt in out source destination
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
Chain OUTPUT (policy ACCEPT 27 packets, 2076 bytes)
pkts bytes target prot opt in out source destination
And $ip addr ls
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether d0:50:99:51:78:7a brd ff:ff:ff:ff:ff:ff
inet *.77/24 brd *.255 scope global eth0
valid_lft forever preferred_lft forever
inet *.167/24 brd *.255 scope global secondary dynamic eth0
valid_lft 24612sec preferred_lft 24612sec
inet6 X::X:X:X:787a/64 scope link
valid_lft forever preferred_lft forever
3: eth1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000
link/ether d0:50:99:51:78:7b brd ff:ff:ff:ff:ff:ff
UPDATE 2:
I failed to mention that in addition to the IP address not matching up, the MAC ID also did not match. This really made me think twice about whether the traffic was indeed coming from my machine. However: (1) unplugging my server from the network made the traffic disappear; (2) move to a different network port and the traffic followed; and (3) tcpdump port 123 showed the aberrant traffic:
13:24:33.329514 IP cumm024-0701-dhcp-233.bu.edu.ntp > 183.61.254.77.44300: NTPv2, Reserved, length 440
13:24:33.329666 IP cumm024-0701-dhcp-233.bu.edu.ntp > 183.61.254.77.44300: NTPv2, Reserved, length 440
13:24:33.329777 IP cumm024-0701-dhcp-233.bu.edu.ntp > 183.61.254.77.44300: NTPv2, Reserved, length 296
UPDATE 3:
$ss -uapn 'sport = :123'
State Recv-Q Send-Q Local Address:Port Peer Address:Port
(i.e., nothing)
$sudo cat /proc/net/dev
Inter-| Receive | Transmit
face |bytes packets errs drop fifo frame compressed multicast|bytes packets errs drop fifo colls carrier compressed
lo: 327357 5455 0 0 0 0 0 0 327357 5455 0 0 0 0 0 0
eth1: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
eth0: 13642399917 36270491 0 6522 0 0 0 2721337 45098276 368537 0 0 0 0 0 0
UPDATE 4: Those packets were typical of a few days ago. Today (but yes, still very high):
20:19:37.011762 IP cumm024-0701-dhcp-233.bu.edu.ntp > 103.56.63.147.26656: NTPv2, Reserved, length 152
20:19:37.011900 IP cumm024-0701-dhcp-233.bu.edu.ntp > 202.83.122.78.58066: NTPv2, Reserved, length 152
20:19:37.012036 IP cumm024-0701-dhcp-233.bu.edu.ntp > 103.56.63.147.17665: NTPv2, Reserved, length 152
20:19:37.014539 IP cumm024-0701-dhcp-233.bu.edu.ntp > 202.83.122.78.27945: NTPv2, Reserved, length 152
20:19:37.015482 IP cumm024-0701-dhcp-233.bu.edu.ntp > 202.83.122.78.42426: NTPv2, Reserved, length 152
20:19:37.015644 IP cumm024-0701-dhcp-233.bu.edu.ntp > 103.56.63.147.16086: NTPv2, Reserved, length 152
$ sudo ss -uapn '( sport = :42426 or dport = :42426 )'
State Recv-Q Send-Q Local Address:Port Peer Address:Port
Yes, I can ping the *.233 IP:
$ping 128.197.112.233
PING 128.197.112.233 (128.197.112.233) 56(84) bytes of data.
64 bytes from 128.197.112.233: icmp_seq=1 ttl=64 time=0.278 ms
64 bytes from 128.197.112.233: icmp_seq=2 ttl=64 time=0.282 ms
64 bytes from 128.197.112.233: icmp_seq=3 ttl=64 time=0.320 ms
No, the MAC don't match My hardware MAC address is: d0:50:99:51:78:7a The traffic is associate with MAC: bc:5f:f4:fe:a1:00
UPDATE 5: As requested, a port scan against *.233:
Starting Nmap 6.00 ( http://nmap.org ) at 2016-12-20 20:38 EET
NSE: Loaded 17 scripts for scanning.
Initiating SYN Stealth Scan at 20:38
Scanning cumm024-0701-dhcp-233.bu.edu (128.197.112.233) [1024 ports]
Discovered open port 22/tcp on 128.197.112.233
Completed SYN Stealth Scan at 20:38, 9.79s elapsed (1024 total ports)
Initiating Service scan at 20:38
Scanning 1 service on cumm024-0701-dhcp-233.bu.edu (128.197.112.233)
Completed Service scan at 20:38, 0.37s elapsed (1 service on 1 host)
Initiating OS detection (try #1) against cumm024-0701-dhcp-233.bu.edu (128.197.112.233)
Initiating Traceroute at 20:38
Completed Traceroute at 20:38, 0.10s elapsed
NSE: Script scanning 128.197.112.233.
[+] Nmap scan report for cumm024-0701-dhcp-233.bu.edu (128.197.112.233)
Host is up (0.083s latency).
Not shown: 1013 filtered ports
PORT STATE SERVICE VERSION
21/tcp closed ftp
22/tcp open ssh OpenSSH 5.5p1 Debian 6+squeeze1 (protocol 2.0)
23/tcp closed telnet
25/tcp closed smtp
43/tcp closed whois
80/tcp closed http
105/tcp closed unknown
113/tcp closed ident
210/tcp closed z39.50
443/tcp closed https
554/tcp closed rtsp
Device type: general purpose
Running: Linux 2.6.X
OS CPE: cpe:/o:linux:kernel:2.6
OS details: DD-WRT v24-sp2 (Linux 2.6.19)
Uptime guess: 45.708 days (since Sat Nov 5 03:39:36 2016)
Network Distance: 9 hops
TCP Sequence Prediction: Difficulty=204 (Good luck!)
IP ID Sequence Generation: All zeros
Service Info: OS: Linux; CPE: cpe:/o:linux:kernel
TRACEROUTE (using port 25/tcp)
HOP RTT ADDRESS
1 0.95 ms router1-lon.linode.com (212.111.33.229)
2 0.70 ms 109.74.207.0
3 1.09 ms be4464.ccr21.lon01.atlas.cogentco.com (204.68.252.85)
4 1.00 ms be2871.ccr42.lon13.atlas.cogentco.com (154.54.58.185)
5 63.45 ms be2983.ccr22.bos01.atlas.cogentco.com (154.54.1.178)
6 63.60 ms TrusteesOfBostonUniversity.demarc.cogentco.com (38.112.23.118)
7 63.55 ms comm595-core-res01-gi2-3-cumm111-bdr-gw01-gi1-2.bu.edu (128.197.254.125)
8 63.61 ms cumm024-dist-aca01-gi5-2-comm595-core-aca01-gi2-2.bu.edu (128.197.254.206)
9 90.28 ms cumm024-0701-dhcp-233.bu.edu (128.197.112.233)
OS and Service detection performed. Please report any incorrect results at http://nmap.org/submit/ .
Nmap done: 1 IP address (1 host up) scanned in 20.73 seconds
Raw packets sent: 557 (25.462KB) | Rcvd: 97 (8.560KB)
and on UDP:
Starting Nmap 6.00 ( http://nmap.org ) at 2016-12-20 20:44 EET
NSE: Loaded 17 scripts for scanning.
Initiating Ping Scan at 20:44
Scanning 128.197.112.233 [4 ports]
Completed Ping Scan at 20:44, 1.10s elapsed (1 total hosts)
Initiating UDP Scan at 20:44
Scanning cumm024-0701-dhcp-233.bu.edu (128.197.112.233) [1024 ports]
Completed UDP Scan at 20:44, 6.31s elapsed (1024 total ports)
Initiating Service scan at 20:44
Scanning 1024 services on cumm024-0701-dhcp-233.bu.edu (128.197.112.233)
Service scan Timing: About 0.39% done
Service scan Timing: About 3.12% done; ETC: 22:12 (1:25:46 remaining)
Service scan Timing: About 6.05% done; ETC: 21:53 (1:04:39 remaining)
Service scan Timing: About 8.98% done; ETC: 21:46 (0:56:03 remaining)
Discovered open port 123/udp on 128.197.112.233
Discovered open|filtered port 123/udp on cumm024-0701-dhcp-233.bu.edu (128.197.112.233) is actually open
Completed Service scan at 21:31, 2833.50s elapsed (1024 services on 1 host)
Initiating OS detection (try #1) against cumm024-0701-dhcp-233.bu.edu (128.197.112.233)
Retrying OS detection (try #2) against cumm024-0701-dhcp-233.bu.edu (128.197.112.233)
NSE: Script scanning 128.197.112.233.
Initiating NSE at 21:31
Completed NSE at 21:31, 10.02s elapsed
[+] Nmap scan report for cumm024-0701-dhcp-233.bu.edu (128.197.112.233)
Host is up (0.089s latency).
Not shown: 1023 open|filtered ports
PORT STATE SERVICE VERSION
123/udp open ntp?
1 service unrecognized despite returning data. If you know the service/version, please submit the following fingerprint at http://www.insecure.org/cgi-bin/servicefp-submit.cgi :
SF-Port123-UDP:V=6.00%I=7%D=12/20%Time=58597D5C%P=x86_64-unknown-linux-gnu
SF:%r(NTPRequest,30,"\xe4\x02\x04\xee\0\0\x8a\xff\0:t\xd9\x84\xa3\x04e\xdb
SF:\xcaeEX\xdbC'\xc5O#Kq\xb1R\xf3\xdc\x03\xfb\xb8\+>U\xab\xdc\x03\xfb\xb8\
SF:+T\xd1\xe9")%r(Citrix,C,"\xde\xc0\x010\x02\0\xa8\xe3\0\0\0\0");
Too many fingerprints match this host to give specific OS details
Network Distance: 9 hops
OS and Service detection performed. Please report any incorrect results at http://nmap.org/submit/ .
Nmap done: 1 IP address (1 host up) scanned in 2863.89 seconds
Raw packets sent: 175 (6.720KB) | Rcvd: 50 (10.088KB)
