# Diagnosing DNS Failures with dig and tcpdump

"It's always DNS" is a running joke for a reason. But "DNS is broken" covers a lot of very different problems: a name that doesn't exist, a server that's down, a firewall in the way, or a client asking the wrong server entirely. Each one needs a different fix, and they can look identical from an application's point of view.

In this post I'll build a small lab, break DNS in five different ways, and show how `dig` and a packet capture tell them apart, usually within a single command.

## Reading dig output: the lines that matter

`dig` prints a lot, but for troubleshooting you only need a few lines. Here's a healthy lookup:

```plaintext
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 3122
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; ANSWER SECTION:
www.lab.example.	0	IN	A	203.0.113.80

;; Query time: 0 msec
;; SERVER: 203.0.113.53#53(203.0.113.53) (UDP)
```

| Line | What to check |
| --- | --- |
| `status:` | The server's verdict: `NOERROR`, `NXDOMAIN`, `REFUSED`, `SERVFAIL` |
| `flags:` / `ANSWER:` | How many records came back, plus flags like `aa` (authoritative) and `tc` (truncated) |
| `SERVER:` | **Which server actually answered**, and whether it was over UDP or TCP |
| `Query time:` | How long it took. Anything in seconds means something waited and retried |

If there's no `status` line at all, the server never answered. That's a different class of problem, and it's where the packet capture comes in.

## Lab setup

Same approach as my [TCP handshake](https://josephoshaughnessy.hashnode.dev/tcp-handshake-wireshark) and [MTU](https://josephoshaughnessy.hashnode.dev/mtu-black-hole) posts: Linux network namespaces on one machine. This time it's a client, a router and a small DNS server running dnsmasq:

![](https://cdn.hashnode.com/uploads/covers/6ad4eef768be819cd453981f/4d5fe546-1cbc-4988-b683-cde6bc16d506.png align="center")

The DNS server answers for a made-up zone, `lab.example`, with three names:

| Name | Record | Why it's there |
| --- | --- | --- |
| `www.lab.example` | A 203.0.113.80 | A normal lookup |
| `mail.lab.example` | A 203.0.113.25 | A second normal lookup |
| `big.lab.example` | Nine TXT records, about 1,450 bytes | Too big for UDP, used in failure 4 |

Grab `dns-lab.sh` and `cleanup.sh` from [GitHub](https://github.com/LostInEther/network-labs/tree/main/dns-failures), then:

```bash
sudo apt install -y dnsmasq-base dnsutils tcpdump
sudo bash dns-lab.sh
```

One detail worth knowing: commands run with `ip netns exec client01` read their DNS settings from `/etc/netns/client01/resolv.conf` rather than `/etc/resolv.conf`. The script points that file at dns01, which is what makes a plain `dig` in client01 use the lab server.

## Failure 1: the server answers "no"

The easiest DNS problems are the ones where the server replies. The network is fine; the answer just isn't what you wanted. The `status` field tells you which kind of "no" it is.

**A typo in the name:**

```bash
sudo ip netns exec client01 dig ww.lab.example
```

```plaintext
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 18747
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1
```

**NXDOMAIN** means the name doesn't exist at all.

**The right name, the wrong record type.** `www` has an IPv4 address but no IPv6 one:

```bash
sudo ip netns exec client01 dig www.lab.example AAAA
```

```plaintext
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 4093
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1
```

This one catches people out. The status is `NOERROR`, but `ANSWER: 0`. That's usually called **NODATA**: the name exists, it just has no record of that type. An app that only asks for AAAA records will fail here even though the A record is fine.

**A name the server won't look up for you:**

```bash
sudo ip netns exec client01 dig www.example.com
```

```plaintext
;; ->>HEADER<<- opcode: QUERY, status: REFUSED, id: 25218
```

**REFUSED** means the server is up and reachable but won't answer that query. My lab server only knows about `lab.example` and has no upstream to forward to, so it refuses everything else. In real networks this is usually a resolver that only allows recursion for certain client subnets, or an authoritative server being asked about a zone it doesn't host.

![](https://cdn.hashnode.com/uploads/covers/6ad4eef768be819cd453981f/1a1b2989-b949-48b5-9e40-97c3a56ff6e9.png align="center")

The good news with all three: the network is working. Stop looking at firewalls and start looking at the DNS data or server config.

## Failure 2: no answer at all

Now a network problem. Drop DNS traffic silently on the router, the way a firewall with a default-deny policy would:

```bash
sudo ip netns exec rtr01 iptables -A FORWARD -p udp --dport 53 -j DROP
```

Start a capture, then run the lookup:

```bash
sudo ip netns exec client01 tcpdump -i c0 -nn -w udpdrop.pcap port 53 &
time sudo ip netns exec client01 dig www.lab.example
```

```plaintext
;; communications error to 203.0.113.53#53: timed out
;; communications error to 203.0.113.53#53: timed out
;; communications error to 203.0.113.53#53: timed out
;; no servers could be reached

real	0m15.040s
```

There's no `status` line, because the server never replied. `dig` tried three times, waiting five seconds each time (its defaults are `+tries=3` and `+timeout=5`), then gave up after 15 seconds.

Applications don't use `dig`. They use the system resolver, which you can test with `getent`:

```bash
time sudo ip netns exec client01 getent hosts www.lab.example
```

![](https://cdn.hashnode.com/uploads/covers/6ad4eef768be819cd453981f/ff439e1d-2425-45f1-aaaa-7dbeb54c5128.png align="center")

That took **20 seconds** to fail in my lab. The capture shows why: it asked for an IPv6 address (AAAA) twice and an IPv4 address (A) twice, five seconds apart, and none of them got an answer.

![](https://cdn.hashnode.com/uploads/covers/6ad4eef768be819cd453981f/ea6dd4be-6b24-40d0-b78a-afa47088f674.png align="center")

That's the signature of a silent drop: queries going out, the same query ID repeated, and nothing coming back. It's also why users report "everything is really slow" rather than "DNS is down": every new connection sits waiting for that timeout.

**Next step:** capture on the DNS server too. If the queries arrive there, the server or its host firewall is ignoring them. If they never arrive, something in the path is dropping them.

Clear the rule before moving on: `sudo ip netns exec rtr01 iptables -F`.

## Failure 3: connection refused

This time the lookup fails instantly:

```plaintext
;; communications error to 203.0.113.53#53: connection refused
;; no servers could be reached
```

DNS normally runs over UDP, which has no connections, so what's being "refused"? Something sent back an **ICMP port unreachable** message. Two very different problems produce that, and you can tell them apart in the capture.

**Cause A: the DNS service isn't running.** Stop dnsmasq on dns01:

```bash
sudo kill $(cat /tmp/dns-lab/dnsmasq.pid)
```

**Cause B: a firewall is rejecting the traffic.** Start dnsmasq again and add a reject rule on the router instead:

```bash
sudo ip netns exec dns01 dnsmasq --conf-file=/tmp/dns-lab/dnsmasq.conf --pid-file=/tmp/dns-lab/dnsmasq.pid
sudo ip netns exec rtr01 iptables -A FORWARD -p udp --dport 53 -j REJECT
```

Capture each one with `tcpdump -i c0 -nn port 53 or icmp` and compare who sent the ICMP message:

| Cause | ICMP port unreachable comes from | TTL |
| --- | --- | --- |
| Service stopped | 203.0.113.53 (the DNS server itself) | 63 |
| Firewall reject | 198.51.100.1 (the router) | 64 |

![](https://cdn.hashnode.com/uploads/covers/6ad4eef768be819cd453981f/32de075c-8bfb-44ec-81e9-8a6744ef213a.png align="center")

Wireshark shows two TTL values on each ICMP packet, such as `64,63`. The first is the TTL of the ICMP message itself, which is the one that tells you who sent it. The second belongs to the copy of your original query that's quoted inside the ICMP message.

If the ICMP comes from the server's own address, the server is up but nothing is listening on port 53: restart or fix the DNS service. If it comes from a device in the path, that device has a reject rule: go and look at its policy. It's the same trick as the TTL check in my [TCP handshake post](https://josephoshaughnessy.hashnode.dev/tcp-handshake-wireshark): the source and TTL of the error tell you who really sent it.

Clear the rule before moving on: `sudo ip netns exec rtr01 iptables -F`.

## Failure 4: small answers work, big ones fail

There's a common myth that DNS is "just UDP 53". It isn't. When an answer is too big to fit in a single UDP response, the server sends back a truncated reply with the **TC (truncated) flag** set, and the client is expected to ask again over **TCP port 53**.

You can see this with the big TXT record in the lab. First, a normal lookup:

```bash
sudo ip netns exec client01 dig big.lab.example TXT
```

```plaintext
;; Truncated, retrying in TCP mode.
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 14662
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 9, AUTHORITY: 0, ADDITIONAL: 1
;; SERVER: 203.0.113.53#53(203.0.113.53) (TCP)
;; MSG SIZE  rcvd: 1449
```

The 1,449-byte answer doesn't fit in the 1,232 bytes `dig` advertises for UDP, so it switched to TCP without being asked. Add `+ignore` to see the truncated UDP reply on its own: `flags: qr aa tc rd ra` and `ANSWER: 0`.

Now block TCP 53 on the router, as a firewall rule that only allows "DNS = UDP 53" would:

```bash
sudo ip netns exec rtr01 iptables -A FORWARD -p tcp --dport 53 -j DROP
```

The big lookup now fails after about 30 seconds:

```plaintext
;; Truncated, retrying in TCP mode.
;; Connection to 203.0.113.53#53(203.0.113.53) for big.lab.example failed: timed out.
;; no servers could be reached
```

But a normal lookup still works instantly:

```bash
sudo ip netns exec client01 dig +short www.lab.example
```

```plaintext
203.0.113.80
```

![](https://cdn.hashnode.com/uploads/covers/6ad4eef768be819cd453981f/934012c6-9180-4151-8643-3e865b2f475c.png align="center")

This is a nasty one in real life, because most lookups work fine. The ones that break are those with big answers: domains with lots of TXT records (SPF, domain verification), DNSSEC-signed responses, and names with many A or AAAA records. It's the DNS version of an MTU problem.

**The fix:** allow **both UDP and TCP port 53** to your DNS servers. [RFC 7766](https://www.rfc-editor.org/rfc/rfc7766) makes TCP support a requirement for DNS, not an optional extra.

Clear the rule before moving on: `sudo ip netns exec rtr01 iptables -F`.

## Failure 5: the client is asking the wrong server

Sometimes the network and the DNS server are both fine, and the client is simply configured to ask somewhere else. Point client01 at an address with no DNS server on it:

```bash
echo "nameserver 203.0.113.99" | sudo tee /etc/netns/client01/resolv.conf
sudo ip netns exec client01 dig +tries=1 +timeout=2 www.lab.example
```

```plaintext
;; communications error to 203.0.113.99#53: timed out
;; no servers could be reached
```

On its own that looks like failure 2. The quickest way to tell them apart is to **ask the server you expect directly**, using `@`:

```bash
sudo ip netns exec client01 dig @203.0.113.53 +short www.lab.example
```

```plaintext
203.0.113.80
```

Asking dns01 directly works, so the network and the server are fine. The problem is the client's resolver config. Two quick checks:

- The `SERVER:` line in `dig`'s output, or the address in the error, shows which server was actually asked.
- `cat /etc/resolv.conf` (or `resolvectl status` on systems running systemd-resolved) shows what the client is configured to use.

![](https://cdn.hashnode.com/uploads/covers/6ad4eef768be819cd453981f/4df18199-ec98-445d-a0a0-88e05c247129.png align="center")

In real networks this is usually a wrong DHCP option, a VPN client that pushed its own DNS servers, or an old static config left behind after a server migration.

Put it back before you finish: `echo "nameserver 203.0.113.53" | sudo tee /etc/netns/client01/resolv.conf`.

## Quick reference

| What dig shows | What it means | Where to look |
| --- | --- | --- |
| `status: NXDOMAIN` | The name doesn't exist | Spelling, or the zone's records |
| `status: NOERROR`, `ANSWER: 0` | NODATA: the name exists, but not that record type | Missing A or AAAA record |
| `status: REFUSED` | The server won't answer that query | Recursion or access settings on the DNS server |
| `timed out` three times, no status | A silent drop; nothing came back | Capture at both ends; firewalls in the path |
| `connection refused` | ICMP port unreachable came back | The ICMP source: the server (service down) or a device in the path (reject rule) |
| `Truncated, retrying in TCP mode`, then `timed out` | UDP works, TCP 53 is blocked | Allow TCP 53 as well as UDP 53 |
| Fails normally, works with `dig @server` | The client is asking the wrong server | `/etc/resolv.conf`, DHCP, VPN settings |

When you're done, remove the lab with `sudo bash cleanup.sh`.

## Wrapping up

When "DNS is broken", read `dig` before anything else. If there's a `status` line, the network is fine and the answer is the problem. If there isn't, capture the traffic and work out whether the query is being dropped, rejected, or sent to the wrong place. And make sure TCP 53 is allowed, because "DNS is just UDP" is the assumption behind a lot of the hardest-to-spot failures.