failed to assign search jobs: there are no searcher nodes ca

Q: What causes "failed to assign search jobs: there are no searcher nodes candidates for these jobs" in quickwit-oss/quickwit?

Retry the placement: this is a race condition; a subsequent call will typically see a consistent cluster snapshot.

Error message

failed to assign search jobs: there are no searcher nodes candidates for these jobs

What it means

In `assign_jobs_inner`, after removing all nodes listed in `excluded_addrs` from the candidate pool, the retained list is empty. This is marked in the code as a belt-and-suspenders branch: exclusion only filters when `excluded_addrs.len() < all_nodes.len()`, so filtering to empty 'should never happen'. Hitting it means the node set changed between the size check and the `retain`, or the exclusion precondition was violated.

Source

Thrown at quickwit/quickwit-search/src/search_job_placer.rs:212

        &self,
        mut jobs: Vec<J>,
        excluded_addrs: &HashSet<SocketAddr>,
        load_aware: bool,
    ) -> anyhow::Result<impl Iterator<Item = (SearchServiceClient, Vec<J>)> + use<J>> {
        let mut all_nodes = self.searcher_pool.pairs();

        if all_nodes.is_empty() {
            bail!(
                "failed to assign search jobs: there are no available searcher nodes in the \
                 cluster"
            );
        }
        if !excluded_addrs.is_empty() && excluded_addrs.len() < all_nodes.len() {
            all_nodes.retain(|(grpc_addr, _)| !excluded_addrs.contains(grpc_addr));

            // This should never happen, but... belt and suspenders policy.
            if all_nodes.is_empty() {
                bail!(
                    "failed to assign search jobs: there are no searcher nodes candidates for \
                     these jobs"
                );
            }
            info!(
                "excluded {} nodes from search job placement, {} remaining",
                excluded_addrs.len(),
                all_nodes.len()
            );
        }
        let mut candidate_nodes: Vec<CandidateNode> = all_nodes
            .into_iter()
            .map(|(grpc_addr, searcher_node)| CandidateNode {
                affinity_id: searcher_node.node_id,
                grpc_addr,
                client: searcher_node.client,
                load: None,
            })

View on GitHub (pinned to a39730c5cd)

Solutions

Retry the placement: this is a race condition; a subsequent call will typically see a consistent cluster snapshot.
Reduce how aggressively failed nodes are excluded so exclusions never cover the whole pool, or cap exclusion count to `all_nodes.len() - 1`.
Investigate searcher flapping: check gossip connectivity and searcher crash logs causing rapid membership churn.
If reproducible, treat as a bug in the placer's consistency check and report/fix the check-rethen-retain race (re-check emptiness against the fresh list).

Example fix

// before: check size, then filter (race window)
if !excluded_addrs.is_empty() && excluded_addrs.len() < all_nodes.len() {
    all_nodes.retain(|(grpc_addr, _)| !excluded_addrs.contains(grpc_addr));
}

// after: filter first, fall back to unfiltered pool if emptied
if !excluded_addrs.is_empty() {
    let filtered: Vec<_> = all_nodes.into_iter()
        .filter(|(addr, _)| !excluded_addrs.contains(addr))
        .collect();
    if !filtered.is_empty() { all_nodes = filtered; }
}

Defensive patterns

Strategy: retry

Validate before calling

// Prefer bounding exclusions at the caller:
let excluded: HashSet<SocketAddr> = failed_nodes
    .into_iter()
    .take(cluster.searcher_count().saturating_sub(1))
    .collect();

Try / catch

for attempt in 0..3 {
    match placer.assign_jobs(jobs.clone(), &excluded).await {
        Ok(p) => return run_search(p).await,
        Err(e) if e.to_string().contains("no searcher nodes candidates") => {
            sleep(backoff(attempt)).await;
            excluded.clear(); // fall back to full pool on next try
        }
        Err(e) => return Err(e),
    }
}

Prevention

Keep exclusion lists bounded so they can never cover the whole pool.
Retry placement with cleared exclusions on this specific error.
Investigate and stabilize node membership churn (gossip connectivity, crashes).
Treat this branch as an invariant: log it loudly if it reproduces, since the code declares it unreachable.

When it happens

Trigger: Calling `assign_jobs()` / `assign_jobs_ignoring_load()` where the excluded-address filtering empties the candidate list — e.g., a concurrent cluster membership change removed nodes between `excluded_addrs.len() < all_nodes.len()` being checked and `all_nodes.retain(...)` executing, causing an internal invariant violation.

Common situations: High node churn (searchers flapping in and out of the Chitchat cluster) while searches carrying exclusion lists (e.g., nodes that previously failed a request) are being placed; race between cluster state refresh and job placement.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

AI-assisted analysis of quickwit-oss/quickwit@a39730c5cd (2026-09-08). Data as JSON: /api/errors/b06ee2ae71cf8de3. Report an issue: GitHub.