8

So I was testing a router and it added some random IPv6 addresses to all the machines on my network, including my DNS server. Somehow those IPs were broadcasted around as valid DNS servers (not sure how as only the real router sends IPv6 RA packets) but long story short, now all my machines are sending DNS queries to an IP address that doesn't exist.

If I restart resolved with systemctl restart systemd-resolved then resolvectl still shows these bogus IPs as valid name servers.

They are listed in /etc/resolv.conf so if I delete them there and restart systemd-resolved it just adds the bogus IPs back in again.

If I look in the logs with journalctl --unit=systemd-resolved then it tells me the bogus IPs are operating in "degraded feature mode" but doesn't tell me where it found those IPs to begin with.

Where is it picking up these wrong IP addresses from?? Is there some cache file I need to delete to make it go back to only using the IPs supplied from the IPv6 router advertisements only?

Malvineous
  • 6,524
  • 5
  • 52
  • 76

3 Answers3

4

After some investigation and a systemd bug report, here is what I discovered.

systemd-resolved gets all its DNS information from systemd-networkd, so focus on systemd-networkd as fixing the rogue server there will flow on into systemd-resolved.

The data is stored in /var/run/systemd/netif/ with one file per interface. This is internal and subject to change so might have moved by the time you read this, however I was able to grep these files for the rogue server and delete the file that had it. When I restarted systemd-networkd, it recreated the deleted file in full.

In my case it recreated the file with the rogue DNS server still listed, which meant it was not being cached by systemd but rather it was still being advertised somewhere on the network.

As it was an IPv6 address, I installed radvd (the IPv6 Router Advertisement daemon) and ran radvdump to show all IPv6 RAs that were arriving on the machine. Sure enough before too long one arrived with the rogue DNS server listed, so I could hunt it down and fix it.

Should this not be an option for you, there are some systemd-networkd options you can use to work around the issue. These must be placed in one of the files where your network is configured (/etc/systemd/network/*.network).

# Don't use DNS servers from DHCP responses received via IPv4 (default is true)
[DHCPv4]
UseDNS=false

# Don't use DNS servers from DHCPv6 responses received via IPv6 (default is true)
[DHCPv6]
UseDNS=false

[IPv6AcceptRA]
# Don't use DNS servers from IPv6 Router Advertisement (RA) messages (default is true)
UseDNS=false
# Don't start a DHCPv6 client when an RA message is received.
DHCPv6Client=false
Malvineous
  • 6,524
  • 5
  • 52
  • 76
  • 1
    in the `IPV6AcceptRA` section, did you intend to write `UseDNS=false`? – nadavwr Aug 24 '21 at 20:55
  • Yes I did, not sure how I missed that, thanks for the correction! – Malvineous Aug 25 '21 at 03:04
  • `/var/run/systemd/netif/` seems to be either an old path or typo. ln my case (fully updated Arch) I found this directory to be `/var/run/systemd/resolve/netif` – Attila123 May 23 '22 at 13:24
  • @Attila123 I have both on my system, the one in my answer is owned by `systemd-networkd` and your one is owned by `systemd-resolved`. I'm not sure what the difference is. – Malvineous May 24 '22 at 08:02
3

You may use this command: sudo systemd-resolve --flush-caches or sudo resolvectl flush-caches (latter command taken from the man page of systemd-resolve)

To verify that flush was sucessfull, use: sudo systemd-resolve --statistics

Sample output:

Cache
  Current Cache Size: 0
          Cache Hits: 101
        Cache Misses: 256

Note also the section of the systemd-resolve manpage about /etc/resolv.conf, which mode are you operating in ?

/etc/resolv.conf

Four modes of handling /etc/resolv.conf (see resolv.conf(5)) are supported:

systemd-resolved maintains the /run/systemd/resolve/stub-resolv.conf file for compatibility with traditional Linux programs. This file may be symlinked from /etc/resolv.conf. This file lists the 127.0.0.53 DNS stub (see above) as the only DNS server. It also contains a list of search domains that are in use by systemd-resolved. The list of search domains is always kept up-to-date. Note that /run/systemd/resolve/stub-resolv.conf should not be used directly by applications, but only through a symlink from /etc/resolv.conf. This file may be symlinked from /etc/resolv.conf in order to connect all local clients that bypass local DNS APIs to systemd-resolved with correct search domains settings. This mode of operation is recommended.

A static file /usr/lib/systemd/resolv.conf is provided that lists the 127.0.0.53 DNS stub (see above) as only DNS server. This file may be symlinked from /etc/resolv.conf in order to connect all local clients that bypass local DNS APIs to systemd-resolved. This file does not contain any search domains.

systemd-resolved maintains the /run/systemd/resolve/resolv.conf file for compatibility with traditional Linux programs. This file may be symlinked from /etc/resolv.conf and is always kept up-to-date, containing information about all known DNS servers. Note the file format's limitations: it does not know a concept of per-interface DNS servers and hence only contains system-wide DNS server definitions. Note that /run/systemd/resolve/resolv.conf should not be used directly by applications, but only through a symlink from /etc/resolv.conf. If this mode of operation is used local clients that bypass any local DNS API will also bypass systemd-resolved and will talk directly to the known DNS servers.

Alternatively, /etc/resolv.conf may be managed by other packages, in which case systemd-resolved will read it for DNS configuration data. In this mode of operation systemd-resolved is consumer rather than provider of this configuration file.

Note that the selected mode of operation for this file is detected fully automatically, depending on whether /etc/resolv.conf is a symlink to /run/systemd/resolve/resolv.conf or lists 127.0.0.53 as DNS server.

thecarpy
  • 3,885
  • 1
  • 16
  • 35
  • 2
    Unfortunately this only flushes the lookup cache and the invalid server IPs still remain. I am running in the mode where `systemd-resolved` controls `resolv.conf` (it's a symlink) however it does not use `127.0.0.53` as the machines in question are already running `bind` as a full nameserver. – Malvineous Apr 25 '19 at 00:52
  • Does this help: https://bbs.archlinux.org/viewtopic.php?id=237862 Basically, in your interface config file, set `[IPV6ACCEPTRA] UseDNS=false` Then restart `systemd-networkd` – thecarpy May 02 '19 at 09:55
  • 1
    It might have helped avoid the initial problem, but there are no more RAs on the network (since the offending router is no longer connected or powered), but the rogue IPs are still present and won't go away. – Malvineous May 04 '19 at 12:05
0

/etc/resolv.conf can not be edited directly. And even if you do, the changes will not take effect even though the appropriate service is restarted. The below steps works for me to remove unwanted DNS settings from /etc/resolv.conf on a Ubuntu 20.04 desktop.

  1. sudo nano /etc/resolvconf/resolv.conf.d/head
  2. Make the desired changes on the nano editor.
  3. Restart the service as appropriate. In my case it was : systemctl stop resolvconf.service;systemctl start resolvconf.service
Binita Bharati
  • 111
  • 2