cilium/cilium · error

cannot forward proxied DNS lookup: %w

Error message

cannot forward proxied DNS lookup: %w

What it means

Forwarding the query to the upstream DNS server failed during p.DNSClients.Exchange (a non-timeout error). The proxy already answered SERVFAIL to the client. Timeouts are handled separately (warn + no error response); only real exchange failures get this error.

Source

Thrown at pkg/fqdn/dnsproxy/proxy.go:1098

		Net:            protocol,
		Dialer:         &dialer,
		Timeout:        ProxyForwardTimeout,
		SingleInflight: false,
	}

	response, _, closer, err := p.DNSClients.Exchange(key, conf, request, targetServerAddrStr)
	defer closer()

	stat.UpstreamTime.End(err == nil)
	if err != nil {
		stat.Err = err
		if stat.IsTimeout() {
			scopedLog.Warn("Timeout waiting for response to forwarded proxied DNS lookup", logfields.Error, err)
			p.NotifyOnDNSMsg(time.Now(), ep, epIPPort, targetServerID, targetServer, requestDetails, protocol, false, &stat)
			return
		}
		scopedLog.Error("Cannot forward proxied DNS lookup", logfields.Error, err)
		stat.Err = fmt.Errorf("cannot forward proxied DNS lookup: %w", err)
		p.NotifyOnDNSMsg(time.Now(), ep, epIPPort, targetServerID, targetServer, requestDetails, protocol, false, &stat)
		p.sendErrorResponse(scopedLog, w, request, false)
		return
	}

	scopedLog.Debug("Received DNS response to proxied lookup", logfields.Response, response)
	stat.Success = true

	stat.ProcessingTime.Start()
	// Extract response details for the successful response path.
	responseDetails, err := ExtractResponseMsgDetails(response)
	if err != nil {
		scopedLog.Error("cannot extract DNS response details", logfields.Error, err)
		stat.Err = fmt.Errorf("cannot extract DNS response details: %w", err)
		stat.ProcessingTime.End(false)
		stat.TotalTime.End(false)
		p.sendErrorResponse(scopedLog, w, request, false)
		return

View on GitHub (pinned to ac7b90affa)

Solutions

1. Check connectivity from the node to the upstream DNS server (port 53 tcp/udp) and any NetworkPolicy/firewall blocking egress.
2. Inspect the wrapped error to distinguish connection refused vs network unreachable vs mark/setsockopt failure.
3. Verify the ipcache entry for the DNS server IP is current; restart or wait for resync if the server recently moved.
4. If setSoMark/control errors appear, check BPF/host routing config and endpoint identity state.
5. Reduce reliance on timeouts: confirm ProxyForwardTimeout suits slow upstreams; timeouts alone do not produce this error.

Example fix

// before: firewall drops egress to 8.8.8.8:53 -> exchange fails
// after: allow egress DNS in policy
- toEndpoints:
  - matchLabels: {"reserved:world"}
toPorts:
- ports: [{port: "53", protocol: ANY}]

// Verify upstream reachability before/at deploy time:
for _, s := range upstreams {
    c := dns.Client{Net: "udp", Timeout: 2 * time.Second}
    if _, _, err := c.Exchange(testQuery, s); err != nil {
        log.Printf("upstream %s unreachable: %v", s, err)
    }
}

func isTimeoutErr(err error) bool {
    var ne net.Error
    return errors.As(err, &ne) && ne.Timeout()
}

if err != nil && !isTimeoutErr(err) {
    // transient network errors can be retried with backoff
    return retryable(fmt.Errorf("forward failed: %w", err))
}

When it happens

Trigger: dns.Client.Exchange with ProxyForwardTimeout fails with a non-timeout error: TCP/UDP connection refused by the upstream, network unreachable, no route to host, socket mark/setSoMark control failure, or a malformed truncated exchange.

Common situations: Upstream DNS server down or firewall dropping egress port 53; the endpoint identity's socket mark being rejected by host networking rules; DNS server IP changed (node-local DNS restart) and ipcache is stale; MTU issues corrupting large UDP responses forcing failed retries.

Understand the failure class

• DNS resolution errors: ENOTFOUND and getaddrinfo failures — how hostname lookups fail and how to debug them.

Related errors

• i/o timeout
• timeout reached waiting for lookup for %s from pod %s to ser
• timeout reached waiting for lookup for %s from pod %s to suc
• Service IP %q not found in nslookup output %q
• timeout reached waiting for service %s (last error: %w)

AI-assisted analysis of cilium/cilium@ac7b90affa (2026-08-31). Data as JSON: /api/errors/12c65421f9003765. Report an issue: GitHub.