{"record":{"id":"11fb1ceb042c87f4","repo":"quickwit-oss/quickwit","slug":"observation-stream-ended","errorCode":null,"errorMessage":"observation stream ended","messagePattern":"observation stream ended","errorType":"exception","errorClass":"anyhow::Error","httpStatus":null,"severity":"error","filePath":"quickwit/quickwit-ingest/src/ingest_v2/helpers.rs","lineNumber":168,"sourceCode":"            return Err((error, None));\n        }\n    };\n    loop {\n        tokio::select! {\n            observation = observation_stream.next() => {\n                match observation {\n                    Some(Ok(observation_message)) => {\n                        if observation_message.status() == status {\n                            return Ok(());\n                        }\n                        last_observation = Some(observation_message);\n                    }\n                    Some(Err(error)) => {\n                        let error = anyhow!(error).context(\"observation stream failed\");\n                        return Err((error, last_observation));\n                    }\n                    None => {\n                        return Err((anyhow!(\"observation stream ended\"), last_observation));\n                    }\n                }\n            }\n            _ = &mut sleep => {\n                let error = anyhow!(\n                    \"timed out while waiting for ingester to transition to status {status} after {}\",\n                    timeout_after.pretty_display(),\n                );\n                return Err((error, last_observation));\n            }\n        }\n    }\n}\n\nfn log_ingester_decommission_failure(\n    error: &anyhow::Error,\n    last_observation: &Option<ObservationMessage>,\n) {","sourceCodeStart":150,"sourceCodeEnd":186,"githubUrl":"https://github.com/quickwit-oss/quickwit/blob/a39730c5cdcd1a4fe798403737ae293999ea21f8/quickwit/quickwit-ingest/src/ingest_v2/helpers.rs#L150-L186","documentation":"The ingester's observation stream terminated normally (returned None) before the ingester ever reported the awaited status. Since the stream should stay open while the service runs, an early end means the ingester closed or dropped the stream without reaching the target state. The helper returns the last observed status for diagnosis.","triggerScenarios":"The observation stream future resolves to None in `wait_for_ingester_status_inner` — the ingester ended the stream (shutdown, panic in the observing actor, service stopped) before emitting a status equal to the requested one.","commonSituations":"Ingester gracefully shutting down while being waited on for Ready; a bug causing the observation actor to stop; node decommission racing with a readiness wait.","solutions":["Check the ingester logs for shutdown/panic messages around the time of the failure.","Retry the wait after confirming the ingester is up; the stream is re-established per call.","If the ingester is intentionally shutting down, cancel the wait rather than waiting for readiness.","If reproducible with a healthy ingester, file a bug: the observation stream should stay open while the service runs."],"exampleFix":"// before: blindly waiting\nwait_for_ingester_status(&mut ingester, Status::Ready, timeout).await?;\n// after: check ingester liveness first\nif !is_ingester_alive(&ingester).await {\n    bail!(\"ingester is down; not waiting for status\");\n}\nwait_for_ingester_status(&mut ingester, Status::Ready, timeout).await?;","handlingStrategy":"try-catch","validationCode":"if !is_ingester_running(&cluster, &ingester_id).await {\n    bail!(\"ingester is shutting down; do not wait for readiness\");\n}","typeGuard":null,"tryCatchPattern":"match wait_for_ingester_status(&mut ingester, Status::Ready, timeout).await {\n    Err(e) if e.to_string().contains(\"observation stream ended\") => {\n        error!(\"ingester closed its observation stream before becoming ready; inspect ingester logs\");\n        return Err(e);\n    }\n    other => other?,\n}","preventionTips":["Check ingester logs for shutdowns/panics whenever this error appears.","Do not decommission or stop an ingester concurrently with readiness waits.","Treat a repeatedly-ended stream with a healthy ingester as a bug to report."],"tags":["grpc","stream","ingester","lifecycle"],"backgroundTag":"unexpected-response-shape","analyzedSha":"a39730c5cdcd1a4fe798403737ae293999ea21f8","analyzedAt":"2026-09-08T13:19:37.784Z","contentChangedAt":"2026-09-08T13:19:37.784Z","schemaVersion":2},"datasetVersion":"2026-09-14T11:17:12.474Z"}