{"record":{"id":"bed2fe761856ccf8","repo":"OpenNHP/opennhp","slug":"nhp-server-timeout","errorCode":null,"errorMessage":"NHP Server timeout","messagePattern":"NHP Server timeout","errorType":"http","errorClass":null,"httpStatus":504,"severity":"error","filePath":"endpoints/relay/relay.go","lineNumber":1127,"sourceCode":"\t\t\tudpTimeout, innerCounter, realAddr, cr.id)\n\t\thttp.Error(w, \"relay overloaded\", http.StatusServiceUnavailable)\n\t\treturn\n\t}\n\n\t// Wait for the raw encrypted ACK/COK packet from the server.\n\tselect {\n\tcase rawBytes := <-responseCh:\n\t\tlog.Info(\"[Relay] received response for inner counter=%d, %d raw bytes, forwarding to client %s (server %s)\",\n\t\t\tinnerCounter, len(rawBytes), realAddr, cr.id)\n\n\t\tw.Header().Set(\"Content-Type\", \"application/octet-stream\")\n\t\tw.WriteHeader(http.StatusOK)\n\t\t_, _ = w.Write(rawBytes)\n\n\tcase <-time.After(time.Duration(udpTimeout) * time.Millisecond):\n\t\tlog.Warning(\"[Relay] timeout waiting for server response (inner counter=%d, client %s, server %s)\",\n\t\t\tinnerCounter, realAddr, cr.id)\n\t\thttp.Error(w, \"NHP Server timeout\", http.StatusGatewayTimeout)\n\t}\n}\n\n// realClientAddr returns the originating address of an HTTP request as a\n// *net.UDPAddr so it can be encoded in the RelayForwardMsg.\n//\n// When the direct TCP peer is on the loopback interface — i.e. a local\n// reverse proxy (nginx, etc.) forwarded the request — the proxy's view\n// of the real client is taken from X-Real-IP, which the proxy is\n// expected to overwrite unconditionally (e.g. nginx\n// `proxy_set_header X-Real-IP $remote_addr;`).\n//\n// X-Forwarded-For is intentionally NOT consulted: nginx's standard\n// `$proxy_add_x_forwarded_for` *appends* to whatever XFF the client\n// sent, so its first entry is attacker-controlled. Trusting XFF would\n// let any HTTP client choose the SourceAddr that flows to nhp-server\n// and ultimately to the AC ipset rule, defeating the per-source-IP\n// authorization model.","sourceCodeStart":1109,"sourceCodeEnd":1145,"githubUrl":"https://github.com/OpenNHP/opennhp/blob/6e04ca5ff03222a699c24205cd4bf8fee9af7ffe/endpoints/relay/relay.go#L1109-L1145","documentation":"After the relay successfully enqueues the NHP_RLY forward to the NHP server, it waits on a response channel for the raw encrypted ACK/COK packet. If no response arrives within udpTimeout milliseconds (rs.config.UDPTimeoutMs, default defaultUDPTimeoutMs), handleRelay returns HTTP 504 Gateway Timeout with body 'NHP Server timeout'. This mirrors an upstream-timeout semantic: the relay accepted the request but the NHP server never answered in time.","triggerScenarios":"A forward was sent (rs.sendMsgCh accepted the packet) but the server's ACK/COK never reached connectionRoutine within udpTimeout: (1) UDP packet loss between relay and server, (2) the NHP server is slow or hung, (3) server responds with a counter that fails the pendingRequests ambiguity check so the response is never dispatched to responseCh, (4) UDPTimeoutMs configured too small for the server's real latency.","commonSituations":"Deploying the relay in a region with poor connectivity to the NHP server; server-side processing slowdown under load (e.g. AC firewall operations taking longer than the timeout); clock/latency sensitivity after lowering UDPTimeoutMs for faster failure detection; duplicate inner counters from a misbehaving client causing responses to be dropped by the ambiguity check.","solutions":["Verify the NHP server is healthy and responding to knocks directly (bypass the relay) to isolate where the delay is.","Increase UDPTimeoutMs in the relay's config.toml if legitimate server latency exceeds the current timeout.","Check network path relay->server for UDP loss/MTU issues; NHP runs over encrypted UDP which some middleboxes drop.","Check relay logs for 'ambiguity check' or counter-dispatch warnings indicating the response arrived but was not routed to this waiter.","Implement client retry on 504, ideally with a fresh inner counter, since the original transaction may have completed server-side."],"exampleFix":"// before: single attempt, hard failure on 504\nresp, err := http.Post(relayURL, \"application/octet-stream\", body)\nif resp.StatusCode == http.StatusGatewayTimeout {\n    return fmt.Errorf(\"NHP Server timeout\")\n}\n\n// after: retry with fresh transaction on timeout\nfor attempt := 0; attempt < 2; attempt++ {\n    body = buildKnockWithFreshCounter()\n    resp, err := http.Post(relayURL, \"application/octet-stream\", body)\n    if err == nil && resp.StatusCode == http.StatusOK {\n        return nil\n    }\n}\nreturn fmt.Errorf(\"NHP server did not respond in time after retries\")","handlingStrategy":"retry","validationCode":"// probe server responsiveness before sending real knocks\nfunc serverResponds(relayURL string, timeout time.Duration) bool {\n    client := http.Client{Timeout: timeout}\n    resp, err := client.Post(relayURL, \"application/octet-stream\", probePayload())\n    if err != nil { return false }\n    resp.Body.Close()\n    return resp.StatusCode == http.StatusOK\n}","typeGuard":"func isNHPTimeout(err error) bool {\n    var httpErr *HTTPStatusError\n    if errors.As(err, &httpErr) {\n        return httpErr.StatusCode == http.StatusGatewayTimeout &&\n            strings.Contains(httpErr.Body, \"NHP Server timeout\")\n    }\n    return false\n}","tryCatchPattern":"resp, err := client.Do(req)\nif err != nil { return err }\nif resp.StatusCode == http.StatusGatewayTimeout {\n    // treat as inconclusive: the server may still have processed the knock\n    if isRetryable(payload) {\n        return retryWithFreshCounter(payload)\n    }\n    return ErrUpstreamTimeout\n}","preventionTips":["Set UDPTimeoutMs above the p99 server response latency with headroom","Watch for UDP packet loss between relay and server (NHP runs over UDP) and avoid lossy paths","Retry timeouts with a fresh inner counter to avoid duplicate-counter rejection","Alert on '[Relay] timeout waiting for server response' warnings","Keep client counter usage unique per in-flight request to avoid the relay's ambiguity-check drop path"],"tags":["http","relay","timeout","udp","upstream"],"backgroundTag":"request-timeout","analyzedSha":"6e04ca5ff03222a699c24205cd4bf8fee9af7ffe","analyzedAt":"2026-09-07T15:44:59.941Z","contentChangedAt":"2026-09-07T15:44:59.941Z","schemaVersion":2},"datasetVersion":"2026-09-16T04:17:20.429Z"}