{"record":{"id":"3f1e165bfd470e9b","repo":"OpenNHP/opennhp","slug":"server-instance-busy","errorCode":null,"errorMessage":"server instance busy","messagePattern":"server instance busy","errorType":"http","errorClass":null,"httpStatus":503,"severity":"warning","filePath":"endpoints/relay/relay.go","lineNumber":1025,"sourceCode":"\t\tinst = cr.pickInstance()\n\t}\n\tif inst == nil {\n\t\thttp.Error(w, \"server has no usable instance\", http.StatusServiceUnavailable)\n\t\treturn\n\t}\n\n\t// Bound concurrent forwards (and thus pendingRequests size) per\n\t// instance. Non-blocking acquire: if the instance is saturated, shed\n\t// the request with 503 rather than queueing — the alternative is the\n\t// pending map growing unbounded when an adversary opens faster than\n\t// the 5s handler timeout drains. Released on handler return.\n\tselect {\n\tcase inst.inFlight <- struct{}{}:\n\t\tdefer func() { <-inst.inFlight }()\n\tdefault:\n\t\tlog.Warning(\"[Relay] instance %s at in-flight cap (%d); shedding forward from %s\",\n\t\t\tinst.addr, MaxInFlightPerInstance, r.RemoteAddr)\n\t\thttp.Error(w, \"server instance busy\", http.StatusServiceUnavailable)\n\t\treturn\n\t}\n\n\tlog.Info(\"[Relay] forwarding %d-byte inner packet (counter=%d, server=%s) from client %s to %s (sticky=%v)\",\n\t\tn, innerCounter, cr.id, realAddr, inst.addr, cr.sticky)\n\n\t// Register a pending request under (counter, realAddr) on the instance.\n\t// The connection routine dispatches the server's ACK/COK to this channel\n\t// only if this handler is the sole waiter on this counter — see the\n\t// ambiguity check in connectionRoutine above.\n\tresponseCh := make(chan []byte, 1)\n\tinst.pendingMu.Lock()\n\twaiters, ok := inst.pendingRequests[innerCounter]\n\tif !ok {\n\t\twaiters = make(map[string]chan []byte)\n\t\tinst.pendingRequests[innerCounter] = waiters\n\t}\n\tif _, dup := waiters[realAddrKey]; dup {","sourceCodeStart":1007,"sourceCodeEnd":1043,"githubUrl":"https://github.com/OpenNHP/opennhp/blob/6e04ca5ff03222a699c24205cd4bf8fee9af7ffe/endpoints/relay/relay.go#L1007-L1043","documentation":"handleRelay responds with HTTP 503 'server instance busy' when the chosen instance's in-flight semaphore (MaxInFlightPerInstance) is already full. The relay deliberately sheds the request with a non-blocking channel acquire instead of queueing, protecting the backend from overload; a warning is logged with the instance address and cap.","triggerScenarios":"POSTing while the target instance already has MaxInFlightPerInstance concurrent forwards in progress — sustained traffic spikes, slow upstream NHP servers holding responses, or many waiters blocked on slow server ACKs.","commonSituations":"Traffic burst exceeding per-instance concurrency cap; backend NHP server slowed (CPU, network, GC) so forwards pile up; cap set too low for real traffic volume; load not balanced across instances (all traffic hashing to one instance).","solutions":["Retry the request after a short backoff — this is an intentional load-shedding signal","Scale out backend NHP server instances so traffic spreads across more slots","Raise MaxInFlightPerInstance if the backend can safely handle more concurrent forwards","Investigate why the upstream server is slow (logs, latency metrics) — slow responses are the usual root cause"],"exampleFix":"// before: cap too small for load\nMaxInFlightPerInstance = 16\n// after\nMaxInFlightPerInstance = 256 // sized from backend load testing\n// plus client-side retry with backoff on 503","handlingStrategy":"retry","validationCode":"// gauge instance load before sending if you control the relay metrics\nif currentInFlight(inst) >= MaxInFlightPerInstance {\n    return errors.New(\"instance saturated; back off\")\n}","typeGuard":null,"tryCatchPattern":"if resp.StatusCode == http.StatusServiceUnavailable {\n    b, _ := io.ReadAll(resp.Body)\n    if strings.Contains(string(b), \"instance busy\") {\n        return retryWithBackoff(send) // honors load shedding; use jittered backoff\n    }\n}","preventionTips":["Implement exponential backoff with jitter on 503s","Size MaxInFlightPerInstance from load tests","Scale out backends when saturation warnings appear in relay logs","Fix slow upstream servers — sustained shedding signals backend slowness"],"tags":["relay","overload","backpressure"],"backgroundTag":"rate-limit-exceeded","analyzedSha":"6e04ca5ff03222a699c24205cd4bf8fee9af7ffe","analyzedAt":"2026-09-07T15:44:59.941Z","contentChangedAt":"2026-09-07T15:44:59.941Z","schemaVersion":2},"datasetVersion":"2026-09-16T04:17:20.429Z"}