quickwit-oss/quickwit · critical

node not found in pending

Error message

node not found in pending

What it means

serve_quickwit spawns the REST server as a named Tokio task and awaits its JoinHandle. The `.expect` fires when the JoinHandle returns Err, meaning the task panicked or was cancelled before completing. The library treats a dead REST server task as an unrecoverable condition, so it panics rather than silently continuing without the REST API.

Solutions

1. Inspect logs immediately before the panic message to find the underlying panic (via the panic hook/backtrace) and fix the root cause in the handler or subsystem.
2. Run with RUST_BACKTRACE=1 to capture a full backtrace identifying which handler or task panicked.
3. Check resource limits (memory, file descriptors) on the host; panics under OOM pressure are common in long-running servers.
4. Validate the Quickwit config with `quickwit config check` so components do not panic during startup inside the REST server task.

Example fix

// before: panic on join error
.expect("tasks running the REST server should not panic or be cancelled")
// after: report and propagate the failure
let rest_result = spawn_named_task(rest_server, "rest_server").await;
let rest = rest_result
    .map_err(|err| anyhow::anyhow!("REST server task failed: {err}"))
    .and_then(|res| res.context("REST server failed"));

// wrap the serve result
match serve_quickwit(config).await {
    Err(err) if err.to_string().contains("REST server") => {
        eprintln!("REST server task died: {err:#}");
        // inspect logs for the original panic before this message
    }
    other => other?,
}

When it happens

Trigger: The `rest_server` future panics while handling a request (e.g. a bug in a handler, or a dependent component such as the metastore or cluster actor panics); or the spawned task is cancelled/aborted and the JoinHandle yields a JoinError.

Common situations: A panic in a request handler or actor the REST server awaits during startup; tokio runtime shutdown while the server task is still running; memory exhaustion or an OOM kill triggered inside the task; running with a config that makes an internally awaited subsystem panic at boot.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

• OTP logs or traces do not support VRL transforms
• 60 should be non-zero
• closure always returns Some
• compactor service enabled but no compaction client available
• `doc_batch` should not be empty

AI-assisted analysis of quickwit-oss/quickwit@a39730c5cd (2026-09-08). Data as JSON: /api/errors/d80b0da4011a13dd. Report an issue: GitHub.

Appendix: source

Thrown at quickwit/quickwit-dst/src/models/parquet_data_model.rs:97

impl DataModelState {
    fn pending_for(&self, node: Node) -> &BTreeSet<DataPoint> {
        for (n, set) in &self.pending {
            if *n == node {
                return set;
            }
        }
        // Should not happen in well-formed model.
        panic!("node not found in pending");
    }

    fn pending_for_mut(&mut self, node: Node) -> &mut BTreeSet<DataPoint> {
        for (n, set) in &mut self.pending {
            if *n == node {
                return set;
            }
        }
        panic!("node not found in pending");
    }

    fn all_stored_rows(&self) -> BTreeSet<DataPoint> {
        self.splits
            .iter()
            .flat_map(|s| s.rows.iter().cloned())
            .collect()
    }

    fn all_pending_rows(&self) -> BTreeSet<DataPoint> {
        self.pending
            .iter()
            .flat_map(|(_, set)| set.iter().cloned())
            .collect()
    }
}

/// Actions.

View on GitHub (pinned to a39730c5cd)