{"record":{"id":"83057233a39fa1b3","repo":"netbirdio/netbird","slug":"signal-receive-stream-stalled","errorCode":null,"errorMessage":"signal receive stream stalled","messagePattern":"signal receive stream stalled","errorType":"console","errorClass":null,"httpStatus":null,"severity":"error","filePath":"shared/signal/client/grpc.go","lineNumber":44,"sourceCode":")\n\nconst (\n\t// receiveInactivityThreshold is how long the receive stream may be silent\n\t// before the watchdog actively probes it. The gRPC transport can stay\n\t// healthy (keepalive satisfied) while the server stops delivering messages,\n\t// which the transport layer cannot detect.\n\treceiveInactivityThreshold = 30 * time.Second\n\t// receiveProbeTimeout is how long the watchdog waits for its self-addressed\n\t// probe to round-trip back on the stream before declaring the receive\n\t// direction dead.\n\treceiveProbeTimeout = 10 * time.Second\n\t// receiveWatchdogInterval is how often the watchdog evaluates the stream.\n\treceiveWatchdogInterval = 10 * time.Second\n)\n\n// errReceiveStreamStalled is reported when the receive stream is transport-alive\n// but no longer delivering messages, so the stream is torn down to reconnect.\nvar errReceiveStreamStalled = errors.New(\"signal receive stream stalled\")\n\n// ConnStateNotifier is a wrapper interface of the status recorder\ntype ConnStateNotifier interface {\n\tMarkSignalDisconnected(error)\n\tMarkSignalConnected()\n}\n\n// GrpcClient Wraps the Signal Exchange Service gRpc client\ntype GrpcClient struct {\n\tkey        wgtypes.Key\n\trealClient proto.SignalExchangeClient\n\tsignalConn *grpc.ClientConn\n\tctx        context.Context\n\tstream     proto.SignalExchange_ConnectStreamClient\n\t// connectedCh used to notify goroutines waiting for the connection to the Signal stream\n\tconnectedCh chan struct{}\n\tmux         sync.Mutex\n\t// StreamConnected indicates whether this client is StreamConnected to the Signal stream","sourceCodeStart":26,"sourceCodeEnd":62,"githubUrl":"https://github.com/netbirdio/netbird/blob/93e97f4bf1ad715072dcb3fb6cdb1763431b5a9c/shared/signal/client/grpc.go#L26-L62","documentation":"errReceiveStreamStalled is raised by the signal client's receive watchdog when the gRPC stream is transport-alive but delivered no messages within receiveInactivityThreshold (30s) and a self-addressed probe failed to round-trip within receiveProbeTimeout (10s). A half-open TCP path can otherwise hang peer signaling forever, so the client tears the stream down and reconnects.","triggerScenarios":"Silent network breakage the transport cannot detect: NAT or firewall idle timeouts dropping the mapping, host suspend/resume, or a path that blackholes the flow while local sends still succeed.","commonSituations":"Long-idle agents behind aggressive NAT/connection-timeout devices; VMs migrated between hosts; laptops resumed after sleep; restrictive middleboxes on the signal path.","solutions":["Let the client's built-in reconnect logic run — the teardown is intentional recovery","If stalls recur, check NAT/firewall idle timeouts on the network path","Keep the agent on a current version so watchdog thresholds match the deployed signal service"],"exampleFix":null,"handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"select {\ncase err := <-watchdogErrs:\n\tif errors.Is(err, errReceiveStreamStalled) {\n\t\t// expected recovery: the client tears down and reconnects; let it, or trigger reconnect sooner\n\t}\n}","preventionTips":["Do not suppress the reconnect the watchdog triggers — it is the fix","If stalls are frequent, inspect NAT/firewall idle timeouts on the path","Keep agent and signal versions aligned so watchdog constants match server behavior"],"tags":["signal","grpc","watchdog","network","reconnection"],"backgroundTag":null,"analyzedSha":"93e97f4bf1ad715072dcb3fb6cdb1763431b5a9c","analyzedAt":"2026-08-16T03:09:19.136Z","schemaVersion":2},"datasetVersion":"2026-08-16T08:17:34.114Z"}