OpenNHP/opennhp · error

relay overloaded

Error message

relay overloaded

What it means

The NHP relay's HTTP handler (handleRelay in endpoints/relay/relay.go) forwards encrypted knock packets to an NHP server over UDP via a bounded send channel (capacity PacketQueueSizePerConnection). When that channel stays full for the full udpTimeout window (rs.config.UDPTimeoutMs, defaulting to defaultUDPTimeoutMs), the handler gives up enqueueing the forward and returns HTTP 503 with body 'relay overloaded'. This is deliberate backpressure: the naked channel send would otherwise block the HTTP goroutine forever under a slow upstream.

Solutions

1. Check the target NHP server instance is reachable and draining UDP traffic (logs show 'send queue full'); restart or fix the server.
2. Reduce request concurrency or add client-side rate limiting / retries with backoff on 503 responses.
3. Increase PacketQueueSizePerConnection or raise UDPTimeoutMs in the relay's config to tolerate bursts.
4. Deploy multiple relay server instances and verify instance selection/load balancing spreads traffic.
5. Monitor rs.sendMsgCh depth and the '[Relay] send queue full' error rate to size capacity correctly.

Example fix

// before: client fails hard on 503
resp, err := http.Post(relayURL, "application/octet-stream", body)
if resp.StatusCode != http.StatusOK { return err }

// after: bounded retry with backoff on 503
for attempt := 0; attempt < 3; attempt++ {
    resp, err := http.Post(relayURL, "application/octet-stream", body)
    if err == nil && resp.StatusCode == http.StatusOK {
        return nil
    }
    time.Sleep(time.Duration(1<<attempt) * 100 * time.Millisecond)
}
return fmt.Errorf("relay overloaded after retries")

// before sending, check relay health endpoint / recent success rate
func relayHealthy(healthURL string) bool {
    resp, err := http.Get(healthURL)
    if err != nil { return false }
    defer resp.Body.Close()
    return resp.StatusCode == http.StatusOK
}

func isRelayOverloaded(err error) bool {
    var httpErr *HTTPStatusError
    if errors.As(err, &httpErr) {
        return httpErr.StatusCode == http.StatusServiceUnavailable &&
            strings.Contains(httpErr.Body, "relay overloaded")
    }
    return false
}

resp, err := client.Do(req)
if err != nil { return err }
if resp.StatusCode == http.StatusServiceUnavailable {
    // back off and retry with jitter
    time.Sleep(backoff(attempt))
    return retryForward(payload)
}

When it happens

Trigger: Calling the relay's HTTP forward endpoint when the UDP send queue to the selected NHP server instance is saturated: (1) the upstream NHP server is down or unreachable so sendMessageRoutine cannot drain the queue, (2) request rate exceeds the relay's UDP send throughput, (3) UDPTimeoutMs is set too low relative to burst traffic so the select's time.After fires before the channel has room.

Common situations: Load testing or traffic spikes overwhelming a single relay->server connection; an NHP server behind a bad network/firewall dropping UDP so packets queue up; multiple test routing requests (like the listed TestRouting_*) hammering handleRelay concurrently while no server drains the queue; misconfigured UDPTimeoutMs after a config change.

Related errors

• rkn rate limit exceeded
• server instance busy
• NHP Server timeout
• relay source address
• inner packet too large

AI-assisted analysis of OpenNHP/opennhp@6e04ca5ff0 (2026-09-07). Data as JSON: /api/errors/f02c84bbdc8bf00d. Report an issue: GitHub.

Appendix: source

Thrown at endpoints/relay/relay.go:1110

		udpTimeout = defaultUDPTimeoutMs
	}

	// Hand the message to sendMessageRoutine. A naked send would block
	// indefinitely if the channel (capacity PacketQueueSizePerConnection)
	// is full — net/http's WriteTimeout closes the TCP connection but
	// does not unblock a goroutine parked on a channel send, so a slow
	// upstream server would silently leak handler goroutines under load.
	// Bound the wait by the same UDP timeout used for the response.
	select {
	case rs.sendMsgCh <- md:
	case <-r.Context().Done():
		log.Warning("[Relay] client disconnected before send queued (counter=%d, client %s, server %s)",
			innerCounter, realAddr, cr.id)
		return
	case <-time.After(time.Duration(udpTimeout) * time.Millisecond):
		log.Error("[Relay] send queue full for %dms, dropping forward (counter=%d, client %s, server %s)",
			udpTimeout, innerCounter, realAddr, cr.id)
		http.Error(w, "relay overloaded", http.StatusServiceUnavailable)
		return
	}

	// Wait for the raw encrypted ACK/COK packet from the server.
	select {
	case rawBytes := <-responseCh:
		log.Info("[Relay] received response for inner counter=%d, %d raw bytes, forwarding to client %s (server %s)",
			innerCounter, len(rawBytes), realAddr, cr.id)

		w.Header().Set("Content-Type", "application/octet-stream")
		w.WriteHeader(http.StatusOK)
		_, _ = w.Write(rawBytes)

	case <-time.After(time.Duration(udpTimeout) * time.Millisecond):
		log.Warning("[Relay] timeout waiting for server response (inner counter=%d, client %s, server %s)",
			innerCounter, realAddr, cr.id)
		http.Error(w, "NHP Server timeout", http.StatusGatewayTimeout)
	}

View on GitHub (pinned to 6e04ca5ff0)