{"record":{"id":"12c65421f9003765","repo":"cilium/cilium","slug":"cannot-forward-proxied-dns-lookup-w","errorCode":null,"errorMessage":"cannot forward proxied DNS lookup: %w","messagePattern":"cannot forward proxied DNS lookup: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"pkg/fqdn/dnsproxy/proxy.go","lineNumber":1098,"sourceCode":"\t\tNet:            protocol,\n\t\tDialer:         &dialer,\n\t\tTimeout:        ProxyForwardTimeout,\n\t\tSingleInflight: false,\n\t}\n\n\tresponse, _, closer, err := p.DNSClients.Exchange(key, conf, request, targetServerAddrStr)\n\tdefer closer()\n\n\tstat.UpstreamTime.End(err == nil)\n\tif err != nil {\n\t\tstat.Err = err\n\t\tif stat.IsTimeout() {\n\t\t\tscopedLog.Warn(\"Timeout waiting for response to forwarded proxied DNS lookup\", logfields.Error, err)\n\t\t\tp.NotifyOnDNSMsg(time.Now(), ep, epIPPort, targetServerID, targetServer, requestDetails, protocol, false, &stat)\n\t\t\treturn\n\t\t}\n\t\tscopedLog.Error(\"Cannot forward proxied DNS lookup\", logfields.Error, err)\n\t\tstat.Err = fmt.Errorf(\"cannot forward proxied DNS lookup: %w\", err)\n\t\tp.NotifyOnDNSMsg(time.Now(), ep, epIPPort, targetServerID, targetServer, requestDetails, protocol, false, &stat)\n\t\tp.sendErrorResponse(scopedLog, w, request, false)\n\t\treturn\n\t}\n\n\tscopedLog.Debug(\"Received DNS response to proxied lookup\", logfields.Response, response)\n\tstat.Success = true\n\n\tstat.ProcessingTime.Start()\n\t// Extract response details for the successful response path.\n\tresponseDetails, err := ExtractResponseMsgDetails(response)\n\tif err != nil {\n\t\tscopedLog.Error(\"cannot extract DNS response details\", logfields.Error, err)\n\t\tstat.Err = fmt.Errorf(\"cannot extract DNS response details: %w\", err)\n\t\tstat.ProcessingTime.End(false)\n\t\tstat.TotalTime.End(false)\n\t\tp.sendErrorResponse(scopedLog, w, request, false)\n\t\treturn","sourceCodeStart":1080,"sourceCodeEnd":1116,"githubUrl":"https://github.com/cilium/cilium/blob/ac7b90affa4baf0642e6685319d56907b3a73a6d/pkg/fqdn/dnsproxy/proxy.go#L1080-L1116","documentation":"Forwarding the query to the upstream DNS server failed during p.DNSClients.Exchange (a non-timeout error). The proxy already answered SERVFAIL to the client. Timeouts are handled separately (warn + no error response); only real exchange failures get this error.","triggerScenarios":"dns.Client.Exchange with ProxyForwardTimeout fails with a non-timeout error: TCP/UDP connection refused by the upstream, network unreachable, no route to host, socket mark/setSoMark control failure, or a malformed truncated exchange.","commonSituations":"Upstream DNS server down or firewall dropping egress port 53; the endpoint identity's socket mark being rejected by host networking rules; DNS server IP changed (node-local DNS restart) and ipcache is stale; MTU issues corrupting large UDP responses forcing failed retries.","solutions":["Check connectivity from the node to the upstream DNS server (port 53 tcp/udp) and any NetworkPolicy/firewall blocking egress.","Inspect the wrapped error to distinguish connection refused vs network unreachable vs mark/setsockopt failure.","Verify the ipcache entry for the DNS server IP is current; restart or wait for resync if the server recently moved.","If setSoMark/control errors appear, check BPF/host routing config and endpoint identity state.","Reduce reliance on timeouts: confirm ProxyForwardTimeout suits slow upstreams; timeouts alone do not produce this error."],"exampleFix":"// before: firewall drops egress to 8.8.8.8:53 -> exchange fails\n// after: allow egress DNS in policy\n- toEndpoints:\n  - matchLabels: {\"reserved:world\"}\ntoPorts:\n- ports: [{port: \"53\", protocol: ANY}]","handlingStrategy":"retry","validationCode":"// Verify upstream reachability before/at deploy time:\nfor _, s := range upstreams {\n    c := dns.Client{Net: \"udp\", Timeout: 2 * time.Second}\n    if _, _, err := c.Exchange(testQuery, s); err != nil {\n        log.Printf(\"upstream %s unreachable: %v\", s, err)\n    }\n}","typeGuard":"func isTimeoutErr(err error) bool {\n    var ne net.Error\n    return errors.As(err, &ne) && ne.Timeout()\n}","tryCatchPattern":"if err != nil && !isTimeoutErr(err) {\n    // transient network errors can be retried with backoff\n    return retryable(fmt.Errorf(\"forward failed: %w\", err))\n}","preventionTips":["Allow egress port 53 (tcp+udp) to upstream DNS servers in NetworkPolicy/firewall.","Monitor upstream DNS health with periodic probes.","Keep ProxyForwardTimeout larger than the upstream's typical latency.","Watch ipcache sync for DNS server IPs after node-local DNS changes."],"tags":["dns","network","upstream","timeout"],"backgroundTag":"dns-upstream-forward-failed","analyzedSha":"ac7b90affa4baf0642e6685319d56907b3a73a6d","analyzedAt":"2026-08-31T18:27:15.868Z","schemaVersion":2},"datasetVersion":"2026-08-31T19:17:28.585Z"}