{"record":{"id":"f02c84bbdc8bf00d","repo":"OpenNHP/opennhp","slug":"relay-overloaded","errorCode":null,"errorMessage":"relay overloaded","messagePattern":"relay overloaded","errorType":"http","errorClass":null,"httpStatus":503,"severity":"error","filePath":"endpoints/relay/relay.go","lineNumber":1110,"sourceCode":"\t\tudpTimeout = defaultUDPTimeoutMs\n\t}\n\n\t// Hand the message to sendMessageRoutine. A naked send would block\n\t// indefinitely if the channel (capacity PacketQueueSizePerConnection)\n\t// is full — net/http's WriteTimeout closes the TCP connection but\n\t// does not unblock a goroutine parked on a channel send, so a slow\n\t// upstream server would silently leak handler goroutines under load.\n\t// Bound the wait by the same UDP timeout used for the response.\n\tselect {\n\tcase rs.sendMsgCh <- md:\n\tcase <-r.Context().Done():\n\t\tlog.Warning(\"[Relay] client disconnected before send queued (counter=%d, client %s, server %s)\",\n\t\t\tinnerCounter, realAddr, cr.id)\n\t\treturn\n\tcase <-time.After(time.Duration(udpTimeout) * time.Millisecond):\n\t\tlog.Error(\"[Relay] send queue full for %dms, dropping forward (counter=%d, client %s, server %s)\",\n\t\t\tudpTimeout, innerCounter, realAddr, cr.id)\n\t\thttp.Error(w, \"relay overloaded\", http.StatusServiceUnavailable)\n\t\treturn\n\t}\n\n\t// Wait for the raw encrypted ACK/COK packet from the server.\n\tselect {\n\tcase rawBytes := <-responseCh:\n\t\tlog.Info(\"[Relay] received response for inner counter=%d, %d raw bytes, forwarding to client %s (server %s)\",\n\t\t\tinnerCounter, len(rawBytes), realAddr, cr.id)\n\n\t\tw.Header().Set(\"Content-Type\", \"application/octet-stream\")\n\t\tw.WriteHeader(http.StatusOK)\n\t\t_, _ = w.Write(rawBytes)\n\n\tcase <-time.After(time.Duration(udpTimeout) * time.Millisecond):\n\t\tlog.Warning(\"[Relay] timeout waiting for server response (inner counter=%d, client %s, server %s)\",\n\t\t\tinnerCounter, realAddr, cr.id)\n\t\thttp.Error(w, \"NHP Server timeout\", http.StatusGatewayTimeout)\n\t}","sourceCodeStart":1092,"sourceCodeEnd":1128,"githubUrl":"https://github.com/OpenNHP/opennhp/blob/6e04ca5ff03222a699c24205cd4bf8fee9af7ffe/endpoints/relay/relay.go#L1092-L1128","documentation":"The NHP relay's HTTP handler (handleRelay in endpoints/relay/relay.go) forwards encrypted knock packets to an NHP server over UDP via a bounded send channel (capacity PacketQueueSizePerConnection). When that channel stays full for the full udpTimeout window (rs.config.UDPTimeoutMs, defaulting to defaultUDPTimeoutMs), the handler gives up enqueueing the forward and returns HTTP 503 with body 'relay overloaded'. This is deliberate backpressure: the naked channel send would otherwise block the HTTP goroutine forever under a slow upstream.","triggerScenarios":"Calling the relay's HTTP forward endpoint when the UDP send queue to the selected NHP server instance is saturated: (1) the upstream NHP server is down or unreachable so sendMessageRoutine cannot drain the queue, (2) request rate exceeds the relay's UDP send throughput, (3) UDPTimeoutMs is set too low relative to burst traffic so the select's time.After fires before the channel has room.","commonSituations":"Load testing or traffic spikes overwhelming a single relay->server connection; an NHP server behind a bad network/firewall dropping UDP so packets queue up; multiple test routing requests (like the listed TestRouting_*) hammering handleRelay concurrently while no server drains the queue; misconfigured UDPTimeoutMs after a config change.","solutions":["Check the target NHP server instance is reachable and draining UDP traffic (logs show 'send queue full'); restart or fix the server.","Reduce request concurrency or add client-side rate limiting / retries with backoff on 503 responses.","Increase PacketQueueSizePerConnection or raise UDPTimeoutMs in the relay's config to tolerate bursts.","Deploy multiple relay server instances and verify instance selection/load balancing spreads traffic.","Monitor rs.sendMsgCh depth and the '[Relay] send queue full' error rate to size capacity correctly."],"exampleFix":"// before: client fails hard on 503\nresp, err := http.Post(relayURL, \"application/octet-stream\", body)\nif resp.StatusCode != http.StatusOK { return err }\n\n// after: bounded retry with backoff on 503\nfor attempt := 0; attempt < 3; attempt++ {\n    resp, err := http.Post(relayURL, \"application/octet-stream\", body)\n    if err == nil && resp.StatusCode == http.StatusOK {\n        return nil\n    }\n    time.Sleep(time.Duration(1<<attempt) * 100 * time.Millisecond)\n}\nreturn fmt.Errorf(\"relay overloaded after retries\")","handlingStrategy":"retry","validationCode":"// before sending, check relay health endpoint / recent success rate\nfunc relayHealthy(healthURL string) bool {\n    resp, err := http.Get(healthURL)\n    if err != nil { return false }\n    defer resp.Body.Close()\n    return resp.StatusCode == http.StatusOK\n}","typeGuard":"func isRelayOverloaded(err error) bool {\n    var httpErr *HTTPStatusError\n    if errors.As(err, &httpErr) {\n        return httpErr.StatusCode == http.StatusServiceUnavailable &&\n            strings.Contains(httpErr.Body, \"relay overloaded\")\n    }\n    return false\n}","tryCatchPattern":"resp, err := client.Do(req)\nif err != nil { return err }\nif resp.StatusCode == http.StatusServiceUnavailable {\n    // back off and retry with jitter\n    time.Sleep(backoff(attempt))\n    return retryForward(payload)\n}","preventionTips":["Load-test the relay at expected peak knock rates and size PacketQueueSizePerConnection accordingly","Monitor '[Relay] send queue full' log rate and alert on sustained occurrences","Keep UDPTimeoutMs consistent with observed server drain latency","Spread traffic across multiple relay instances","Ensure the upstream NHP server is always reachable before routing traffic through the relay"],"tags":["http","relay","overload","backpressure","udp"],"backgroundTag":"rate-limit-exceeded","analyzedSha":"6e04ca5ff03222a699c24205cd4bf8fee9af7ffe","analyzedAt":"2026-09-07T15:44:59.941Z","contentChangedAt":"2026-09-07T15:44:59.941Z","schemaVersion":2},"datasetVersion":"2026-09-16T04:17:20.429Z"}