quickwit-oss/quickwit · error · anyhow::Error

observation stream ended

Error message

observation stream ended

What it means

The ingester's observation stream terminated normally (returned None) before the ingester ever reported the awaited status. Since the stream should stay open while the service runs, an early end means the ingester closed or dropped the stream without reaching the target state. The helper returns the last observed status for diagnosis.

Source

Thrown at quickwit/quickwit-ingest/src/ingest_v2/helpers.rs:168

            return Err((error, None));
        }
    };
    loop {
        tokio::select! {
            observation = observation_stream.next() => {
                match observation {
                    Some(Ok(observation_message)) => {
                        if observation_message.status() == status {
                            return Ok(());
                        }
                        last_observation = Some(observation_message);
                    }
                    Some(Err(error)) => {
                        let error = anyhow!(error).context("observation stream failed");
                        return Err((error, last_observation));
                    }
                    None => {
                        return Err((anyhow!("observation stream ended"), last_observation));
                    }
                }
            }
            _ = &mut sleep => {
                let error = anyhow!(
                    "timed out while waiting for ingester to transition to status {status} after {}",
                    timeout_after.pretty_display(),
                );
                return Err((error, last_observation));
            }
        }
    }
}

fn log_ingester_decommission_failure(
    error: &anyhow::Error,
    last_observation: &Option<ObservationMessage>,
) {

View on GitHub (pinned to a39730c5cd)

Solutions

1. Check the ingester logs for shutdown/panic messages around the time of the failure.
2. Retry the wait after confirming the ingester is up; the stream is re-established per call.
3. If the ingester is intentionally shutting down, cancel the wait rather than waiting for readiness.
4. If reproducible with a healthy ingester, file a bug: the observation stream should stay open while the service runs.

Example fix

// before: blindly waiting
wait_for_ingester_status(&mut ingester, Status::Ready, timeout).await?;
// after: check ingester liveness first
if !is_ingester_alive(&ingester).await {
    bail!("ingester is down; not waiting for status");
}
wait_for_ingester_status(&mut ingester, Status::Ready, timeout).await?;

if !is_ingester_running(&cluster, &ingester_id).await {
    bail!("ingester is shutting down; do not wait for readiness");
}

match wait_for_ingester_status(&mut ingester, Status::Ready, timeout).await {
    Err(e) if e.to_string().contains("observation stream ended") => {
        error!("ingester closed its observation stream before becoming ready; inspect ingester logs");
        return Err(e);
    }
    other => other?,
}

When it happens

Trigger: The observation stream future resolves to None in `wait_for_ingester_status_inner` — the ingester ended the stream (shutdown, panic in the observing actor, service stopped) before emitting a status equal to the requested one.

Common situations: Ingester gracefully shutting down while being waited on for Ready; a bug causing the observation actor to stop; node decommission racing with a readiness wait.

Related errors

• observation stream failed
• actor `{}` is disconnected
• timed out while waiting for ingester to transition to status
• position of a Kafka partition should never be EOF
• position of a Kinesis shard should never be EOF

AI-assisted analysis of quickwit-oss/quickwit@a39730c5cd (2026-09-08). Data as JSON: /api/errors/11fb1ceb042c87f4. Report an issue: GitHub.