quickwit-oss/quickwit · error · anyhow::Error

timed out while waiting for ingester to transition to status

Error message

timed out while waiting for ingester to transition to status {status} after {}

What it means

`wait_for_ingester_status_inner` opens an observation stream to an ingester and waits for it to report the desired status, guarded by a timeout. If the sleep branch of the select fires first, the wait failed: the ingester never reached the requested status (Ready or ReadyForCopy) within the deadline. Callers attach context about which ingester and which status was expected.

Source

Thrown at quickwit/quickwit-ingest/src/ingest_v2/helpers.rs:146

    timeout_after: Duration,
) -> Result<(), (anyhow::Error, Option<ObservationMessage>)> {
    debug_assert!(
        timeout_after > Duration::ZERO,
        "timeout_after should be greater than zero"
    );
    let mut last_observation: Option<ObservationMessage> = None;

    let sleep = tokio::time::sleep(timeout_after);
    tokio::pin!(sleep);

    let mut observation_stream = tokio::select! {
        result = ingester.open_observation_stream(OpenObservationStreamRequest {}) => {
            result
                .context("failed to open observation stream")
                .map_err(|error| (error, None))?
        }
        _ = &mut sleep => {
            let error = anyhow!(
                "timed out while waiting for ingester to transition to status {status} after {}",
                timeout_after.pretty_display(),
            );
            return Err((error, None));
        }
    };
    loop {
        tokio::select! {
            observation = observation_stream.next() => {
                match observation {
                    Some(Ok(observation_message)) => {
                        if observation_message.status() == status {
                            return Ok(());
                        }
                        last_observation = Some(observation_message);
                    }
                    Some(Err(error)) => {
                        let error = anyhow!(error).context("observation stream failed");

View on GitHub (pinned to a39730c5cd)

Solutions

Increase the timeout passed to `wait_for_ingester_status`/`wait_for_ingester_decommission` for slow starts or large WAL replays.
Inspect the ingester logs/metrics to find why it never reached the target status (WAL replay loop, missing control-plane connectivity).
Verify cluster connectivity between the caller and the ingester's gRPC port.
Retry the operation once the ingester is healthy; for decommission, ensure indexing tasks can be rescheduled.

Example fix

// before
wait_for_ingester_status(&mut ingester, ingester_status::Status::Ready, Duration::from_secs(5)).await?;
// after
wait_for_ingester_status(&mut ingester, ingester_status::Status::Ready, Duration::from_secs(60)).await?;

Defensive patterns

Strategy: retry

Validate before calling

// before waiting, confirm the ingester responds at all
let health = ingester.check_health().await
    .context("ingester unreachable; fix connectivity before waiting for status")?;

Try / catch

match wait_for_ingester_status(&mut ingester, Status::Ready, timeout).await {
    Err(e) if e.to_string().contains("timed out while waiting") => {
        warn!(error = %e, "ingester not ready in time; retrying with longer timeout");
        wait_for_ingester_status(&mut ingester, Status::Ready, timeout * 4).await?;
    }
    other => other?,
}

Prevention

Budget timeouts generously for slow ingester starts and large WAL replays.
Monitor ingester readiness metrics during rollouts.
Ensure the control plane can schedule tasks before requesting status transitions.

When it happens

Trigger: Calling `wait_for_ingester_status` or `wait_for_ingester_decommission` while the target ingester keeps reporting a different status until `timeout_after` elapses — e.g. an ingester stuck starting up, still replaying WAL, or never reacting to a decommission request.

Common situations: Ingester pod slow to start (large WAL replay) during cluster rollout; decommissioning a node whose ingester can't drain because control-plane can't place its indexing tasks; network partition making the observation stream stale.

Understand the failure class

Background: Request timed out: what client-side request timeouts mean across libraries (Request timed out, TIMED_OUT, APITimeoutError) — this error's family across 39 libraries.

Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

AI-assisted analysis of quickwit-oss/quickwit@a39730c5cd (2026-09-08). Data as JSON: /api/errors/aa139a1f48f844ed. Report an issue: GitHub.