quickwit-oss/quickwit · critical

failed to assign search jobs: there are no available searche

Error message

failed to assign search jobs: there are no available searcher nodes in the cluster

What it means

The search job placer (`SearchJobPlacer::assign_jobs_inner`) distributes search jobs across the searcher nodes registered in the cluster's searcher pool. This error is raised when the pool is completely empty: `searcher_pool.pairs()` returned no `(grpc_addr, searcher)` entries, so there is nowhere to route any job. It is a fail-fast guard before rendezvous-hash placement is attempted.

Source

Thrown at quickwit/quickwit-search/src/search_job_placer.rs:202

    /// but starts from a uniform zero existing load.
    pub async fn assign_jobs_ignoring_load<J: Job>(
        &self,
        jobs: Vec<J>,
        excluded_addrs: &HashSet<SocketAddr>,
    ) -> anyhow::Result<impl Iterator<Item = (SearchServiceClient, Vec<J>)> + use<J>> {
        self.assign_jobs_inner(jobs, excluded_addrs, false).await
    }

    async fn assign_jobs_inner<J: Job>(
        &self,
        mut jobs: Vec<J>,
        excluded_addrs: &HashSet<SocketAddr>,
        load_aware: bool,
    ) -> anyhow::Result<impl Iterator<Item = (SearchServiceClient, Vec<J>)> + use<J>> {
        let mut all_nodes = self.searcher_pool.pairs();

        if all_nodes.is_empty() {
            bail!(
                "failed to assign search jobs: there are no available searcher nodes in the \
                 cluster"
            );
        }
        if !excluded_addrs.is_empty() && excluded_addrs.len() < all_nodes.len() {
            all_nodes.retain(|(grpc_addr, _)| !excluded_addrs.contains(grpc_addr));

            // This should never happen, but... belt and suspenders policy.
            if all_nodes.is_empty() {
                bail!(
                    "failed to assign search jobs: there are no searcher nodes candidates for \
                     these jobs"
                );
            }
            info!(
                "excluded {} nodes from search job placement, {} remaining",
                excluded_addrs.len(),
                all_nodes.len()

View on GitHub (pinned to a39730c5cd)

Solutions

1. Verify searcher nodes are running and joined the cluster: check `GET /health/live` and cluster membership via `GET /cluster` for searcher nodes with the Searcher role enabled.
2. Start or restart searcher-capable nodes (ensure `searcher` is enabled in node config) and confirm they register in the searcher pool.
3. If searchers are running, check network connectivity / gossip ports between nodes so membership converges.
4. Add retry logic in the client path: searcher nodes leaving transiently (redeploy) is recoverable once they rejoin.
5. Check quickwit logs for searcher departures (graceful shutdown or failure) preceding the error.

Example fix

// before: searching right after cluster start without checking readiness
let hits = client.search(request).await?;

// after: wait until at least one searcher is available
wait_until(|| {
    let state = cluster_snapshot();
    !state.searcher_nodes().is_empty()
}).await?;
let hits = client.search(request).await?;

let searchers: Vec<_> = cluster.members_with_role(Role::Searcher);
if searchers.is_empty() {
    return Err("no searcher nodes available; retry after searchers join");
}

fn has_searcher_nodes(pairs: &[(SocketAddr, SearcherNode)]) -> bool { !pairs.is_empty() }

match placer.assign_jobs(jobs, &excluded).await {
    Ok(placement) => run_search(placement).await,
    Err(e) if e.to_string().contains("no available searcher nodes") => {
        tokio::time::sleep(Duration::from_secs(2)).await;
        retry_with_backoff().await
    }
    Err(e) => return Err(e),
}

When it happens

Trigger: Calling `SearchJobPlacer::assign_jobs()` or `assign_jobs_ignoring_load()` with a non-empty job list while the cluster's searcher pool has zero registered searcher nodes (e.g., all searchers left the Chitchat cluster, no searcher ever joined, or searchers were marked unavailable/dead in the cluster state).

Common situations: All searcher processes are down or restarting during a rolling upgrade; a networking/partition issue removed searchers from the cluster; the searcher service was never deployed in this Quickwit deployment mode (e.g., an indexer/ metastore-only node serving queries); cluster membership (Chitchat gossip) not yet converged right after startup.

Understand the failure class

Background: EmptyResultError / "no results found": when an API or scraper succeeds but returns zero rows — this error's family across 9 libraries.

Related errors

• failed to assign search jobs: there are no searcher nodes ca
• `assign_jobs` should return at least one client or fail.
• unknown tokenizer `{}` for field `{}`
• invalid named document. there are more than 1 value associat
• the `{key}` value has to be a json object

AI-assisted analysis of quickwit-oss/quickwit@a39730c5cd (2026-09-08). Data as JSON: /api/errors/acbd444d6433f62b. Report an issue: GitHub.