The ndots:5 Tax: Why Your Kubernetes Cluster Is Silently Multiplying Your DNS Latency

You probably think your microservices are slow because of inefficient database queries, unoptimized JSON serialization, or perhaps just "noisy neighbors" on your cloud provider's hardware. You might have even spent weeks tuning your JVM or Go garbage collection to shave off 20 milliseconds of tail latency. But for many Kubernetes users, a massive chunk of their network latency has nothing to do with their application code. It’s a hidden tax imposed by a single line in a configuration file you likely never look at.

Kubernetes networking defaults are optimized for convenience, not for speed. Specifically, the default DNS resolution logic—governed by the ndots:5 setting—is a silent performance killer that forces your cluster to perform up to five failed DNS lookups for every single outbound request your application makes.

If you're calling an external API like api.stripe.com, your pod isn't just asking "Where is Stripe?" It's asking "Is Stripe in my namespace? No? Is it in my service cluster? No? Is it in the default namespace? No?" by the time it actually finds the answer, your request has already paid a heavy price in round-trip times and CoreDNS overhead.

The Anatomy of a DNS Lookup in Kubernetes

When you run a pod in Kubernetes, the container runtime injects a /etc/resolv.conf file into it. This file tells the Linux DNS resolver how to handle domain names. If you exec into any running pod and look at that file, you'll see something that looks like this:

# Inside a standard Kubernetes pod
cat /etc/resolv.conf

nameserver 10.96.0.10
search my-namespace.svc.cluster.local svc.cluster.local cluster.local
options ndots:5

The search line defines the "search path"—a list of suffixes the resolver should append to a hostname if it looks like a relative name. The ndots:5 line is the culprit. It tells the resolver: "If a domain name has fewer than 5 dots in it, treat it as a relative name and try appending all the search paths first before trying it as an absolute name."

Think about that. api.stripe.com has two dots. According to the ndots:5 rule, that is a relative name.

The Sequence of Failure

Let’s trace exactly what happens when your application tries to reach api.twitter.com. Because api.twitter.com has only two dots, the resolver goes down the search list in order.

Here is the actual sequence of DNS queries hitting your CoreDNS instances:

1. api.twitter.com.my-namespace.svc.cluster.local -> NXDOMAIN (Non-Existent Domain)
2. api.twitter.com.svc.cluster.local -> NXDOMAIN
3. api.twitter.com.cluster.local -> NXDOMAIN
4. api.twitter.com -> SUCCESS (Finally)

For every single outbound connection, you are generating three useless requests that must be processed by CoreDNS. If your application is high-throughput or makes many external calls, you aren't just slowing down your own app; you are potentially redlining your CoreDNS CPU and causing it to throttle or drop requests for other services in the cluster.

I’ve seen production clusters where 80% of the DNS traffic was just failed NXDOMAIN noise generated by this exact behavior.

Why "5" is the Magic (and Dangerous) Number

You might wonder why the Kubernetes maintainers chose 5. It wasn't an arbitrary number picked out of a hat. It was designed to ensure that internal service discovery works seamlessly across different layers of the Kubernetes hierarchy.

Consider a service named database in the namespace prod. Kubernetes allows you to address this service in several ways:
- database (local namespace)
- database.prod
- database.prod.svc
- database.prod.svc.cluster.local

To make database.prod.svc.cluster.local resolve correctly even when you just type database, the resolver needs a high ndots value. If they had set ndots:1, then database.prod (one dot) would be treated as an absolute name, and the resolver would immediately try to find prod as a Top-Level Domain (TLD) on the open internet. It would fail, and it would never try the search paths that would lead it to the internal service.

By setting it to 5, Kubernetes ensures that almost any internal name you use will eventually be matched against the search paths. But we are paying for that internal convenience with external performance.

Visualizing the Latency Tax

Let’s look at some real output. I’ll use dig inside a pod to simulate how the resolver behaves, but I'll use the +trace and +search flags to show the internal logic.

# Simulating the search path behavior with ndots:5 logic
# Note: standard 'dig' doesn't always honor resolv.conf exactly 
# like glibc's getaddrinfo, but we can see the attempts.

$ time dig +search google.com

; <<>> DiG 9.16.1-Ubuntu <<>> +search google.com
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 54321
;; ... google.com.my-ns.svc.cluster.local ...

;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 54322
;; ... google.com.svc.cluster.local ...

;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 54323
;; ... google.com ...

If your CoreDNS latency is 2ms per request, that google.com lookup just took 6ms-8ms before the connection even started. In a world of microservices where one user request might trigger ten downstream API calls, you are suddenly adding 60ms-100ms of "DNS tax" to your total response time.

Strategy 1: The "Trailing Dot" Hack

The fastest, zero-config way to fix this in your application code is to use Fully Qualified Domain Names (FQDNs). In the DNS world, an absolute name ends with a dot.

If you change your configuration from https://api.stripe.com to https://api.stripe.com., you are telling the DNS resolver: "This is an absolute name. Do not look at the search paths. Go straight to the root."

import requests

# BAD: Triggers the ndots search path traversal
requests.get("https://api.stripe.com/v1/charges")

# GOOD: Bypasses the search path entirely
requests.get("https://api.stripe.com./v1/charges")

Most modern HTTP libraries and language runtimes handle the trailing dot just fine. However, some TLS implementations might get grumpy if the hostname in the certificate doesn't match the one with the trailing dot (though this is becoming rarer). It’s a bit of a "gross" fix, but it's incredibly effective for quick wins.

Strategy 2: Manually Tuning dnsConfig

Kubernetes allows you to override the DNS settings on a per-pod basis. If you have a service that primarily talks to external APIs and doesn't do much internal service discovery (or uses full internal names), you can lower the ndots value in your Deployment manifest.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api-gateway
spec:
  template:
    spec:
      dnsConfig:
        options:
          - name: ndots
            value: "1"
      containers:
        - name: gateway
          image: my-gateway:latest

Warning: If you set ndots: 1, you break short-name service discovery. If your app tries to connect to http://auth-service, it will fail. You will be forced to use the full name: http://auth-service.my-namespace.svc.cluster.local.

For most production-grade systems, using full internal names is actually a better practice anyway, as it's more explicit and avoids ambiguity between namespaces.

Strategy 3: NodeLocal DNSCache

If you can't easily change your application code or your ndots config, you should look at NodeLocal DNSCache.

Normally, every DNS request travels over the network (using iptables or ipvs DNAT) to the CoreDNS pods, which might be on a different node. This adds network latency and connection tracking overhead. NodeLocal DNSCache runs a tiny DNS caching agent on every single node as a DaemonSet.

When your pod pays the "ndots tax," it’s at least paying it to a process running on the same piece of silicon. It doesn't eliminate the multiple lookups, but it makes each lookup significantly faster (sub-millisecond) and takes the load off your central CoreDNS instances.

To check if it’s running in your cluster:

kubectl get pods -n kube-system -l k8s-app=node-local-dns

If you don't see anything, you're missing out on a standard performance optimization.

The Alpine Linux and musl Gotcha

There is a specific edge case involving Alpine Linux that makes the ndots issue even worse. Most Linux distributions (Ubuntu, Debian, CentOS) use glibc for their standard C library. glibc performs DNS lookups sequentially. It tries the first search path, waits for a failure, then tries the second.

Alpine Linux uses musl. To be "efficient," musl sends out all DNS queries in the search path simultaneously.

While this sounds like it would be faster (parallelism!), it often creates a massive race condition and floods CoreDNS with even more traffic. If you are seeing weird DNS intermittent failures or "Name not resolved" errors specifically in Alpine-based containers, the ndots parallel lookup behavior is likely hitting a connection tracking limit in your Linux kernel's conntrack table.

The fix here is usually to use a glibc-based base image (like debian-slim or wolfi) or to be extremely diligent about using the trailing dot hack.

CoreDNS Rewrites: The Nuclear Option

If you have a very specific external domain that is being hit thousands of times a second, you can configure CoreDNS to intercept those requests before they even trigger the search path logic.

In your Corefile:

.:53 {
    errors
    health
    ready
    # Rewrite internal search path attempts for specific external domains
    rewrite name regex api\.stripe\.com\.(.*)\.svc\.cluster\.local api.stripe.com
    kubernetes cluster.local in-addr.arpa ip6.arpa {
       pods insecure
       fallthrough in-addr.arpa ip6.arpa
    }
    prometheus :9153
    forward . /etc/resolv.conf
    cache 30
    loop
    reload
    loadbalance
}

This is a bit of a maintenance nightmare, as you’re hardcoding application logic into your infrastructure config, but for critical paths, it can be a lifesaver.

Measuring the Impact

Don't just take my word for it. You should measure the DNS overhead in your own cluster. One of the best ways to do this is by looking at the CoreDNS metrics via Prometheus.

Look for the coredns_dns_responses_total metric and filter by rcode.

# Percentage of NXDOMAIN responses vs Successful ones
sum(rate(coredns_dns_responses_total{rcode="NXDOMAIN"}[5m])) 
/ 
sum(rate(coredns_dns_responses_total{rcode="NOERROR"}[5m]))

If your ratio of NXDOMAIN to NOERROR is higher than 1:1, you are paying a massive DNS tax. In some poorly optimized clusters, I've seen this ratio go as high as 10:1. That means for every successful name resolution, the infrastructure had to process ten failures. That is pure waste.

Choosing the Right Path

So, how should you handle the ndots tax? It depends on where you are in your Kubernetes journey.

1. For new projects: Start with ndots:1 in your pod templates and train your team to use FQDNs for internal service discovery. It’s cleaner, faster, and more explicit.
2. For existing large-scale projects: Implement NodeLocal DNSCache immediately. It’s the most transparent way to reduce the latency penalty without touching application code.
3. For specific high-latency calls: Use the trailing dot (api.stripe.com.) for external API clients in your code. It’s a 1-character change that can shave 10ms off every outbound request.

DNS is often treated as "magical plumbing" that just works. But in Kubernetes, that plumbing is convoluted by design. Understanding the ndots search logic isn't just a bit of networking trivia—it's a fundamental part of scaling a cluster that is actually performant, rather than just "functional." Stop paying the tax and start being intentional about how your services find the world.