How does DNS work in Kubernetes?

Question

I'm having issue with DNS resolve in Kubernetes. It's quite complex, please be patient to read.

ENV:

• On-premise and private Cloud

OS

• Ubuntu Server 20.04 LTS

System Network:

• VPN s2s
• Custom DNS (10.4.0.149) added to LAN Interface

Kubernetes network:

• Calico
• CoreDNS and NodelocalDNS

Currently K8s Pod on private cloud needs to resolve domain from DNS located on on-premise. I have followed this guide to add custom DNS to help pod reaches to DNS server when resolve domain:

https://kubernetes.io/docs/tasks/administer-cluster/dns-custom-nameservers/

with below block:

projects.com:53 {
    errors
    log
    cache 30
    forward . 10.4.0.149
}

When I do dig from a pod, sometime it is able to resolve domain but sometime it's not. But nslookup works every times. I have also tried to ping domain from pod but it happens the same as dig. Sometime works and sometime not. By the way I have looked at network packet by tcpdump from node the pod located in but no luck.

So the question is how DNS resolve works in Kubernetes? AFAIK the DNS resolve will works with request from pod -> nodelocaldns -> coredns -> /etc/resolv.conf on node. But it seems didn't work like it should.

Could you please advise?

Thanks in advance!

---

I got this from tcpdump:

2503    12.026282   10.244.14.63    169.254.25.10   DNS 105 Standard query 0x91f8 A id-test.projects.com OPT
2506    12.049959   169.254.25.10   10.244.14.63    DNS 105 Standard query response 0x91f8 Format error A id-test.projects.com OPT
2772    14.056998   10.244.14.63    169.254.25.10   DNS 105 Standard query 0xc532 A id-test.projects.com OPT
2785    14.079098   169.254.25.10   10.244.14.63    DNS 105 Standard query response 0xc532 Format error A id-test.projects.com OPT
3355    16.086108   10.244.14.63    169.254.25.10   DNS 105 Standard query 0xf86c A id-test.projects.com OPT
3358    16.108111   169.254.25.10   10.244.14.63    DNS 105 Standard query response 0xf86c Format error A id-test.projects.com OPT
4073    18.115002   10.244.14.63    169.254.25.10   DNS 105 Standard query 0x2ba6 A id-test.projects.com OPT
4126    18.137460   169.254.25.10   10.244.14.63    DNS 105 Standard query response 0x2ba6 Format error A id-test.projects.com OPT
4179    18.250697   10.244.14.64    169.254.25.10   DNS 107 Standard query 0x678f A id-test.projects.com.master.svc.cluster.local
4180    18.250736   10.244.14.64    169.254.25.10   DNS 107 Standard query 0xbc8b AAAA id-test.projects.com.master.svc.cluster.local
4196    18.251611   169.254.25.10   10.244.14.64    DNS 200 Standard query response 0xbc8b No such name AAAA id-test.projects.com.master.svc.cluster.local SOA ns.dns.cluster.local
4202    18.251957   169.254.25.10   10.244.14.64    DNS 200 Standard query response 0x678f No such name A id-test.projects.com.master.svc.cluster.local SOA ns.dns.cluster.local
4203    18.252034   10.244.14.64    169.254.25.10   DNS 100 Standard query 0x360a A id-test.projects.com.svc.cluster.local
4204    18.252060   10.244.14.64    169.254.25.10   DNS 100 Standard query 0x6e09 AAAA id-test.projects.com.svc.cluster.local
4215    18.252524   169.254.25.10   10.244.14.64    DNS 193 Standard query response 0x360a No such name A id-test.projects.com.svc.cluster.local SOA ns.dns.cluster.local
4220    18.252637   169.254.25.10   10.244.14.64    DNS 193 Standard query response 0x6e09 No such name AAAA id-test.projects.com.svc.cluster.local SOA ns.dns.cluster.local
4221    18.252697   10.244.14.64    169.254.25.10   DNS 96  Standard query 0x6b3d A id-test.projects.com.cluster.local
4222    18.252718   10.244.14.64    169.254.25.10   DNS 96  Standard query 0x103f AAAA id-test.projects.com.cluster.local
4233    18.253085   169.254.25.10   10.244.14.64    DNS 189 Standard query response 0x103f No such name AAAA id-test.projects.com.cluster.local SOA ns.dns.cluster.local
4238    18.253281   169.254.25.10   10.244.14.64    DNS 189 Standard query response 0x6b3d No such name A id-test.projects.com.cluster.local SOA ns.dns.cluster.local
4239    18.253350   10.244.14.64    169.254.25.10   DNS 82  Standard query 0x4118 A id-test.projects.com
4240    18.253364   10.244.14.64    169.254.25.10   DNS 82  Standard query 0x151e AAAA id-test.projects.com
4244    18.275928   169.254.25.10   10.244.14.64    DNS 168 Standard query response 0x151e AAAA id-test.projects.com SOA ad.global.com
4246    18.276120   169.254.25.10   10.244.14.64    DNS 118 Standard query response 0x4118 A id-test.projects.com A 10.7.22.15
4632    20.146523   10.244.14.63    169.254.25.10   DNS 105 Standard query 0x5ee1 A id-test.projects.com OPT
4633    20.146709   169.254.25.10   10.244.14.63    DNS 141 Standard query response 0x5ee1 A id-test.projects.com A 10.7.22.15 OPT
5042    22.152642   10.244.14.63    169.254.25.10   DNS 105 Standard query 0x921b A id-test.projects.com OPT
5043    22.152818   169.254.25.10   10.244.14.63    DNS 141 Standard query response 0x921b A id-test.projects.com A 10.7.22.15 OPT
5328    24.159482   10.244.14.63    169.254.25.10   DNS 105 Standard query 0xc555 A id-test.projects.com OPT
5329    24.159653   169.254.25.10   10.244.14.63    DNS 141 Standard query response 0xc555 A id-test.projects.com A 10.7.22.15 OPT
5659    26.165768   10.244.14.63    169.254.25.10   DNS 105 Standard query 0xf88f A id-test.projects.com OPT

I don't know why I got format error when use dig, please pay attention at IP 10.244.14.63, it is curl pod, another (10.244.14.64) is from application service's pod which needs to resolve DNS, when it resolved DNS successfully, the dig command immediately return correct IP for domain name.

What do you think about this issue?

Please advise.

Thanks in advance.