quickwit-oss/quickwit · error · anyhow::Error

observation stream failed

Error message

observation stream failed

What it means

The observation gRPC stream used to track an ingester's status returned an error mid-stream. The helper wraps the underlying tonic error with this context so the caller knows the status watch failed rather than ended normally. The last observed status is preserved to help diagnose how far the ingester got.

Source

Thrown at quickwit/quickwit-ingest/src/ingest_v2/helpers.rs:164

            let error = anyhow!(
                "timed out while waiting for ingester to transition to status {status} after {}",
                timeout_after.pretty_display(),
            );
            return Err((error, None));
        }
    };
    loop {
        tokio::select! {
            observation = observation_stream.next() => {
                match observation {
                    Some(Ok(observation_message)) => {
                        if observation_message.status() == status {
                            return Ok(());
                        }
                        last_observation = Some(observation_message);
                    }
                    Some(Err(error)) => {
                        let error = anyhow!(error).context("observation stream failed");
                        return Err((error, last_observation));
                    }
                    None => {
                        return Err((anyhow!("observation stream ended"), last_observation));
                    }
                }
            }
            _ = &mut sleep => {
                let error = anyhow!(
                    "timed out while waiting for ingester to transition to status {status} after {}",
                    timeout_after.pretty_display(),
                );
                return Err((error, last_observation));
            }
        }
    }
}

View on GitHub (pinned to a39730c5cd)

Solutions

1. Inspect the wrapped tonic error to identify the transport/service failure.
2. Check the ingester's health — it likely crashed or restarted; fix the underlying issue first.
3. Verify network stability and any proxy/LB idle timeouts on gRPC streams.
4. Retry the wait operation; the stream is re-opened on each call.

Example fix

// before
let status = wait_for_ingester_status(&mut ingester, Status::Ready, timeout).await;

// verify the gRPC endpoint before starting a status wait
let channel_ready = tonic_transport_channel_ready(&ingester_endpoint).await;
if !channel_ready { bail!("ingester gRPC channel not ready"); }

match wait_for_ingester_status(&mut ingester, Status::Ready, timeout).await {
    Err(e) if e.to_string().contains("observation stream failed") => {
        warn!(error = %e, "status stream broke; retrying");
        wait_for_ingester_status(&mut ingester, Status::Ready, timeout).await?;
    }
    other => other?,
}

When it happens

Trigger: While draining the `open_observation_stream` in `wait_for_ingester_status_inner`, the stream yields `Some(Err(error))` — e.g. gRPC connection reset, transport error, or the ingester service returning an error on the observation stream.

Common situations: Ingester process crash or restart during the wait; network interruption between nodes; load balancer killing an idle gRPC stream; ingester shutting down mid-decommission.

Understand the failure class

Background: "API error: {status}" and "HTTP 401/403/404/429/5xx" errors: non-2xx HTTP responses explained — this error's family across 27 libraries.

Related errors

• observation stream ended
• no server currently available
• timed out while waiting for ingester to transition to status
• lambda invocation failed: {}
• position of a Kafka partition should never be EOF

AI-assisted analysis of quickwit-oss/quickwit@a39730c5cd (2026-09-08). Data as JSON: /api/errors/7027d30e3995fd24. Report an issue: GitHub.