{"record":{"id":"aa139a1f48f844ed","repo":"quickwit-oss/quickwit","slug":"timed-out-while-waiting-for-ingester-to-transition","errorCode":null,"errorMessage":"timed out while waiting for ingester to transition to status {status} after {}","messagePattern":"timed out while waiting for ingester to transition to status (.+?) after (.+?)","errorType":"exception","errorClass":"anyhow::Error","httpStatus":null,"severity":"error","filePath":"quickwit/quickwit-ingest/src/ingest_v2/helpers.rs","lineNumber":146,"sourceCode":"    timeout_after: Duration,\n) -> Result<(), (anyhow::Error, Option<ObservationMessage>)> {\n    debug_assert!(\n        timeout_after > Duration::ZERO,\n        \"timeout_after should be greater than zero\"\n    );\n    let mut last_observation: Option<ObservationMessage> = None;\n\n    let sleep = tokio::time::sleep(timeout_after);\n    tokio::pin!(sleep);\n\n    let mut observation_stream = tokio::select! {\n        result = ingester.open_observation_stream(OpenObservationStreamRequest {}) => {\n            result\n                .context(\"failed to open observation stream\")\n                .map_err(|error| (error, None))?\n        }\n        _ = &mut sleep => {\n            let error = anyhow!(\n                \"timed out while waiting for ingester to transition to status {status} after {}\",\n                timeout_after.pretty_display(),\n            );\n            return Err((error, None));\n        }\n    };\n    loop {\n        tokio::select! {\n            observation = observation_stream.next() => {\n                match observation {\n                    Some(Ok(observation_message)) => {\n                        if observation_message.status() == status {\n                            return Ok(());\n                        }\n                        last_observation = Some(observation_message);\n                    }\n                    Some(Err(error)) => {\n                        let error = anyhow!(error).context(\"observation stream failed\");","sourceCodeStart":128,"sourceCodeEnd":164,"githubUrl":"https://github.com/quickwit-oss/quickwit/blob/a39730c5cdcd1a4fe798403737ae293999ea21f8/quickwit/quickwit-ingest/src/ingest_v2/helpers.rs#L128-L164","documentation":"`wait_for_ingester_status_inner` opens an observation stream to an ingester and waits for it to report the desired status, guarded by a timeout. If the sleep branch of the select fires first, the wait failed: the ingester never reached the requested status (Ready or ReadyForCopy) within the deadline. Callers attach context about which ingester and which status was expected.","triggerScenarios":"Calling `wait_for_ingester_status` or `wait_for_ingester_decommission` while the target ingester keeps reporting a different status until `timeout_after` elapses — e.g. an ingester stuck starting up, still replaying WAL, or never reacting to a decommission request.","commonSituations":"Ingester pod slow to start (large WAL replay) during cluster rollout; decommissioning a node whose ingester can't drain because control-plane can't place its indexing tasks; network partition making the observation stream stale.","solutions":["Increase the timeout passed to `wait_for_ingester_status`/`wait_for_ingester_decommission` for slow starts or large WAL replays.","Inspect the ingester logs/metrics to find why it never reached the target status (WAL replay loop, missing control-plane connectivity).","Verify cluster connectivity between the caller and the ingester's gRPC port.","Retry the operation once the ingester is healthy; for decommission, ensure indexing tasks can be rescheduled."],"exampleFix":"// before\nwait_for_ingester_status(&mut ingester, ingester_status::Status::Ready, Duration::from_secs(5)).await?;\n// after\nwait_for_ingester_status(&mut ingester, ingester_status::Status::Ready, Duration::from_secs(60)).await?;","handlingStrategy":"retry","validationCode":"// before waiting, confirm the ingester responds at all\nlet health = ingester.check_health().await\n    .context(\"ingester unreachable; fix connectivity before waiting for status\")?;","typeGuard":null,"tryCatchPattern":"match wait_for_ingester_status(&mut ingester, Status::Ready, timeout).await {\n    Err(e) if e.to_string().contains(\"timed out while waiting\") => {\n        warn!(error = %e, \"ingester not ready in time; retrying with longer timeout\");\n        wait_for_ingester_status(&mut ingester, Status::Ready, timeout * 4).await?;\n    }\n    other => other?,\n}","preventionTips":["Budget timeouts generously for slow ingester starts and large WAL replays.","Monitor ingester readiness metrics during rollouts.","Ensure the control plane can schedule tasks before requesting status transitions."],"tags":["ingester","timeout","grpc","cluster"],"backgroundTag":"request-timeout","analyzedSha":"a39730c5cdcd1a4fe798403737ae293999ea21f8","analyzedAt":"2026-09-08T13:19:37.784Z","contentChangedAt":"2026-09-08T13:19:37.784Z","schemaVersion":2},"datasetVersion":"2026-09-14T11:17:12.474Z"}