quickwit-oss/quickwit · critical

node not found in pending

Error message

node not found in pending

What it means

serve_quickwit spawns the REST server as a named Tokio task and awaits its JoinHandle. The `.expect` fires when the JoinHandle returns Err, meaning the task panicked or was cancelled before completing. The library treats a dead REST server task as an unrecoverable condition, so it panics rather than silently continuing without the REST API.

Source

Thrown at quickwit/quickwit-dst/src/models/parquet_data_model.rs:97

impl DataModelState {
    fn pending_for(&self, node: Node) -> &BTreeSet<DataPoint> {
        for (n, set) in &self.pending {
            if *n == node {
                return set;
            }
        }
        // Should not happen in well-formed model.
        panic!("node not found in pending");
    }

    fn pending_for_mut(&mut self, node: Node) -> &mut BTreeSet<DataPoint> {
        for (n, set) in &mut self.pending {
            if *n == node {
                return set;
            }
        }
        panic!("node not found in pending");
    }

    fn all_stored_rows(&self) -> BTreeSet<DataPoint> {
        self.splits
            .iter()
            .flat_map(|s| s.rows.iter().cloned())
            .collect()
    }

    fn all_pending_rows(&self) -> BTreeSet<DataPoint> {
        self.pending
            .iter()
            .flat_map(|(_, set)| set.iter().cloned())
            .collect()
    }
}

/// Actions.

View on GitHub (pinned to a39730c5cd)

Solutions

1. Inspect logs immediately before the panic message to find the underlying panic (via the panic hook/backtrace) and fix the root cause in the handler or subsystem.
2. Run with RUST_BACKTRACE=1 to capture a full backtrace identifying which handler or task panicked.
3. Check resource limits (memory, file descriptors) on the host; panics under OOM pressure are common in long-running servers.
4. Validate the Quickwit config with `quickwit config check` so components do not panic during startup inside the REST server task.

Example fix

// before: panic on join error
.expect("tasks running the REST server should not panic or be cancelled")
// after: report and propagate the failure
let rest_result = spawn_named_task(rest_server, "rest_server").await;
let rest = rest_result
    .map_err(|err| anyhow::anyhow!("REST server task failed: {err}"))
    .and_then(|res| res.context("REST server failed"));

// wrap the serve result
match serve_quickwit(config).await {
    Err(err) if err.to_string().contains("REST server") => {
        eprintln!("REST server task died: {err:#}");
        // inspect logs for the original panic before this message
    }
    other => other?,
}

When it happens

Trigger: The `rest_server` future panics while handling a request (e.g. a bug in a handler, or a dependent component such as the metastore or cluster actor panics); or the spawned task is cancelled/aborted and the JoinHandle yields a JoinError.

Common situations: A panic in a request handler or actor the REST server awaits during startup; tokio runtime shutdown while the server task is still running; memory exhaustion or an OOM kill triggered inside the task; running with a config that makes an internally awaited subsystem panic at boot.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

• OTP logs or traces do not support VRL transforms
• position of a Kafka partition should never be EOF
• position of a Kinesis shard should never be EOF
• `doc_batch` should not be empty
• Unexpected span kind: {}

AI-assisted analysis of quickwit-oss/quickwit@a39730c5cd (2026-09-08). Data as JSON: /api/errors/d80b0da4011a13dd. Report an issue: GitHub.