OpenNHP/opennhp · warning

server instance busy

Error message

server instance busy

What it means

handleRelay responds with HTTP 503 'server instance busy' when the chosen instance's in-flight semaphore (MaxInFlightPerInstance) is already full. The relay deliberately sheds the request with a non-blocking channel acquire instead of queueing, protecting the backend from overload; a warning is logged with the instance address and cap.

Solutions

1. Retry the request after a short backoff — this is an intentional load-shedding signal
2. Scale out backend NHP server instances so traffic spreads across more slots
3. Raise MaxInFlightPerInstance if the backend can safely handle more concurrent forwards
4. Investigate why the upstream server is slow (logs, latency metrics) — slow responses are the usual root cause

Example fix

// before: cap too small for load
MaxInFlightPerInstance = 16
// after
MaxInFlightPerInstance = 256 // sized from backend load testing
// plus client-side retry with backoff on 503

// gauge instance load before sending if you control the relay metrics
if currentInFlight(inst) >= MaxInFlightPerInstance {
    return errors.New("instance saturated; back off")
}

if resp.StatusCode == http.StatusServiceUnavailable {
    b, _ := io.ReadAll(resp.Body)
    if strings.Contains(string(b), "instance busy") {
        return retryWithBackoff(send) // honors load shedding; use jittered backoff
    }
}

When it happens

Trigger: POSTing while the target instance already has MaxInFlightPerInstance concurrent forwards in progress — sustained traffic spikes, slow upstream NHP servers holding responses, or many waiters blocked on slow server ACKs.

Common situations: Traffic burst exceeding per-instance concurrency cap; backend NHP server slowed (CPU, network, GC) so forwards pile up; cap set too low for real traffic volume; load not balanced across instances (all traffic hashing to one instance).

Related errors

• relay overloaded
• rkn rate limit exceeded
• server connection table full
• relay: failed to parse config
• relay: privateKeyBase64 must be set in config

AI-assisted analysis of OpenNHP/opennhp@6e04ca5ff0 (2026-09-07). Data as JSON: /api/errors/3f1e165bfd470e9b. Report an issue: GitHub.

Appendix: source

Thrown at endpoints/relay/relay.go:1025

		inst = cr.pickInstance()
	}
	if inst == nil {
		http.Error(w, "server has no usable instance", http.StatusServiceUnavailable)
		return
	}

	// Bound concurrent forwards (and thus pendingRequests size) per
	// instance. Non-blocking acquire: if the instance is saturated, shed
	// the request with 503 rather than queueing — the alternative is the
	// pending map growing unbounded when an adversary opens faster than
	// the 5s handler timeout drains. Released on handler return.
	select {
	case inst.inFlight <- struct{}{}:
		defer func() { <-inst.inFlight }()
	default:
		log.Warning("[Relay] instance %s at in-flight cap (%d); shedding forward from %s",
			inst.addr, MaxInFlightPerInstance, r.RemoteAddr)
		http.Error(w, "server instance busy", http.StatusServiceUnavailable)
		return
	}

	log.Info("[Relay] forwarding %d-byte inner packet (counter=%d, server=%s) from client %s to %s (sticky=%v)",
		n, innerCounter, cr.id, realAddr, inst.addr, cr.sticky)

	// Register a pending request under (counter, realAddr) on the instance.
	// The connection routine dispatches the server's ACK/COK to this channel
	// only if this handler is the sole waiter on this counter — see the
	// ambiguity check in connectionRoutine above.
	responseCh := make(chan []byte, 1)
	inst.pendingMu.Lock()
	waiters, ok := inst.pendingRequests[innerCounter]
	if !ok {
		waiters = make(map[string]chan []byte)
		inst.pendingRequests[innerCounter] = waiters
	}
	if _, dup := waiters[realAddrKey]; dup {

View on GitHub (pinned to 6e04ca5ff0)