What causes "signal receive stream stalled" in netbirdio/netbird?

Let the client's built-in reconnect logic run — the teardown is intentional recovery

signal receive stream stalled

Error message

signal receive stream stalled

What it means

errReceiveStreamStalled is raised by the signal client's receive watchdog when the gRPC stream is transport-alive but delivered no messages within receiveInactivityThreshold (30s) and a self-addressed probe failed to round-trip within receiveProbeTimeout (10s). A half-open TCP path can otherwise hang peer signaling forever, so the client tears the stream down and reconnects.

Source

Thrown at shared/signal/client/grpc.go:44

)

const (
	// receiveInactivityThreshold is how long the receive stream may be silent
	// before the watchdog actively probes it. The gRPC transport can stay
	// healthy (keepalive satisfied) while the server stops delivering messages,
	// which the transport layer cannot detect.
	receiveInactivityThreshold = 30 * time.Second
	// receiveProbeTimeout is how long the watchdog waits for its self-addressed
	// probe to round-trip back on the stream before declaring the receive
	// direction dead.
	receiveProbeTimeout = 10 * time.Second
	// receiveWatchdogInterval is how often the watchdog evaluates the stream.
	receiveWatchdogInterval = 10 * time.Second
)

// errReceiveStreamStalled is reported when the receive stream is transport-alive
// but no longer delivering messages, so the stream is torn down to reconnect.
var errReceiveStreamStalled = errors.New("signal receive stream stalled")

// ConnStateNotifier is a wrapper interface of the status recorder
type ConnStateNotifier interface {
	MarkSignalDisconnected(error)
	MarkSignalConnected()
}

// GrpcClient Wraps the Signal Exchange Service gRpc client
type GrpcClient struct {
	key        wgtypes.Key
	realClient proto.SignalExchangeClient
	signalConn *grpc.ClientConn
	ctx        context.Context
	stream     proto.SignalExchange_ConnectStreamClient
	// connectedCh used to notify goroutines waiting for the connection to the Signal stream
	connectedCh chan struct{}
	mux         sync.Mutex
	// StreamConnected indicates whether this client is StreamConnected to the Signal stream

View on GitHub (pinned to 93e97f4bf1)

Solutions

Let the client's built-in reconnect logic run — the teardown is intentional recovery
If stalls recur, check NAT/firewall idle timeouts on the network path
Keep the agent on a current version so watchdog thresholds match the deployed signal service

Defensive patterns

Strategy: retry

Try / catch

select {
case err := <-watchdogErrs:
	if errors.Is(err, errReceiveStreamStalled) {
		// expected recovery: the client tears down and reconnects; let it, or trigger reconnect sooner
	}
}

Prevention

Do not suppress the reconnect the watchdog triggers — it is the fix
If stalls are frequent, inspect NAT/firewall idle timeouts on the path
Keep agent and signal versions aligned so watchdog constants match server behavior

When it happens

Trigger: Silent network breakage the transport cannot detect: NAT or firewall idle timeouts dropping the mapping, host suspend/resume, or a path that blackholes the flow while local sends still succeed.

Common situations: Long-idle agents behind aggressive NAT/connection-timeout devices; VMs migrated between hosts; laptops resumed after sleep; restrictive middleboxes on the signal path.

Related errors

AI-assisted analysis of netbirdio/netbird@93e97f4bf1 (2026-08-16). Data as JSON: /api/errors/83057233a39fa1b3. Report an issue: GitHub.