{"record":{"id":"7027d30e3995fd24","repo":"quickwit-oss/quickwit","slug":"observation-stream-failed","errorCode":null,"errorMessage":"observation stream failed","messagePattern":"observation stream failed","errorType":"exception","errorClass":"anyhow::Error","httpStatus":null,"severity":"error","filePath":"quickwit/quickwit-ingest/src/ingest_v2/helpers.rs","lineNumber":164,"sourceCode":"            let error = anyhow!(\n                \"timed out while waiting for ingester to transition to status {status} after {}\",\n                timeout_after.pretty_display(),\n            );\n            return Err((error, None));\n        }\n    };\n    loop {\n        tokio::select! {\n            observation = observation_stream.next() => {\n                match observation {\n                    Some(Ok(observation_message)) => {\n                        if observation_message.status() == status {\n                            return Ok(());\n                        }\n                        last_observation = Some(observation_message);\n                    }\n                    Some(Err(error)) => {\n                        let error = anyhow!(error).context(\"observation stream failed\");\n                        return Err((error, last_observation));\n                    }\n                    None => {\n                        return Err((anyhow!(\"observation stream ended\"), last_observation));\n                    }\n                }\n            }\n            _ = &mut sleep => {\n                let error = anyhow!(\n                    \"timed out while waiting for ingester to transition to status {status} after {}\",\n                    timeout_after.pretty_display(),\n                );\n                return Err((error, last_observation));\n            }\n        }\n    }\n}\n","sourceCodeStart":146,"sourceCodeEnd":182,"githubUrl":"https://github.com/quickwit-oss/quickwit/blob/a39730c5cdcd1a4fe798403737ae293999ea21f8/quickwit/quickwit-ingest/src/ingest_v2/helpers.rs#L146-L182","documentation":"The observation gRPC stream used to track an ingester's status returned an error mid-stream. The helper wraps the underlying tonic error with this context so the caller knows the status watch failed rather than ended normally. The last observed status is preserved to help diagnose how far the ingester got.","triggerScenarios":"While draining the `open_observation_stream` in `wait_for_ingester_status_inner`, the stream yields `Some(Err(error))` — e.g. gRPC connection reset, transport error, or the ingester service returning an error on the observation stream.","commonSituations":"Ingester process crash or restart during the wait; network interruption between nodes; load balancer killing an idle gRPC stream; ingester shutting down mid-decommission.","solutions":["Inspect the wrapped tonic error to identify the transport/service failure.","Check the ingester's health — it likely crashed or restarted; fix the underlying issue first.","Verify network stability and any proxy/LB idle timeouts on gRPC streams.","Retry the wait operation; the stream is re-opened on each call."],"exampleFix":"// before\nlet status = wait_for_ingester_status(&mut ingester, Status::Ready, timeout).await;","handlingStrategy":"retry","validationCode":"// verify the gRPC endpoint before starting a status wait\nlet channel_ready = tonic_transport_channel_ready(&ingester_endpoint).await;\nif !channel_ready { bail!(\"ingester gRPC channel not ready\"); }","typeGuard":null,"tryCatchPattern":"match wait_for_ingester_status(&mut ingester, Status::Ready, timeout).await {\n    Err(e) if e.to_string().contains(\"observation stream failed\") => {\n        warn!(error = %e, \"status stream broke; retrying\");\n        wait_for_ingester_status(&mut ingester, Status::Ready, timeout).await?;\n    }\n    other => other?,\n}","preventionTips":["Keep gRPC keepalive settings tuned so proxies/LBs don't kill idle streams.","Monitor ingester crashes and restarts during waits.","Retry transient transport failures automatically."],"tags":["grpc","stream","network","ingester"],"backgroundTag":"http-error-response","analyzedSha":"a39730c5cdcd1a4fe798403737ae293999ea21f8","analyzedAt":"2026-09-08T13:19:37.784Z","contentChangedAt":"2026-09-08T13:19:37.784Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}