{"record":{"id":"acbd444d6433f62b","repo":"quickwit-oss/quickwit","slug":"failed-to-assign-search-jobs-there-are-no-availab","errorCode":null,"errorMessage":"failed to assign search jobs: there are no available searcher nodes in the cluster","messagePattern":"failed to assign search jobs: there are no available searcher nodes in the cluster","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"quickwit/quickwit-search/src/search_job_placer.rs","lineNumber":202,"sourceCode":"    /// but starts from a uniform zero existing load.\n    pub async fn assign_jobs_ignoring_load<J: Job>(\n        &self,\n        jobs: Vec<J>,\n        excluded_addrs: &HashSet<SocketAddr>,\n    ) -> anyhow::Result<impl Iterator<Item = (SearchServiceClient, Vec<J>)> + use<J>> {\n        self.assign_jobs_inner(jobs, excluded_addrs, false).await\n    }\n\n    async fn assign_jobs_inner<J: Job>(\n        &self,\n        mut jobs: Vec<J>,\n        excluded_addrs: &HashSet<SocketAddr>,\n        load_aware: bool,\n    ) -> anyhow::Result<impl Iterator<Item = (SearchServiceClient, Vec<J>)> + use<J>> {\n        let mut all_nodes = self.searcher_pool.pairs();\n\n        if all_nodes.is_empty() {\n            bail!(\n                \"failed to assign search jobs: there are no available searcher nodes in the \\\n                 cluster\"\n            );\n        }\n        if !excluded_addrs.is_empty() && excluded_addrs.len() < all_nodes.len() {\n            all_nodes.retain(|(grpc_addr, _)| !excluded_addrs.contains(grpc_addr));\n\n            // This should never happen, but... belt and suspenders policy.\n            if all_nodes.is_empty() {\n                bail!(\n                    \"failed to assign search jobs: there are no searcher nodes candidates for \\\n                     these jobs\"\n                );\n            }\n            info!(\n                \"excluded {} nodes from search job placement, {} remaining\",\n                excluded_addrs.len(),\n                all_nodes.len()","sourceCodeStart":184,"sourceCodeEnd":220,"githubUrl":"https://github.com/quickwit-oss/quickwit/blob/a39730c5cdcd1a4fe798403737ae293999ea21f8/quickwit/quickwit-search/src/search_job_placer.rs#L184-L220","documentation":"The search job placer (`SearchJobPlacer::assign_jobs_inner`) distributes search jobs across the searcher nodes registered in the cluster's searcher pool. This error is raised when the pool is completely empty: `searcher_pool.pairs()` returned no `(grpc_addr, searcher)` entries, so there is nowhere to route any job. It is a fail-fast guard before rendezvous-hash placement is attempted.","triggerScenarios":"Calling `SearchJobPlacer::assign_jobs()` or `assign_jobs_ignoring_load()` with a non-empty job list while the cluster's searcher pool has zero registered searcher nodes (e.g., all searchers left the Chitchat cluster, no searcher ever joined, or searchers were marked unavailable/dead in the cluster state).","commonSituations":"All searcher processes are down or restarting during a rolling upgrade; a networking/partition issue removed searchers from the cluster; the searcher service was never deployed in this Quickwit deployment mode (e.g., an indexer/ metastore-only node serving queries); cluster membership (Chitchat gossip) not yet converged right after startup.","solutions":["Verify searcher nodes are running and joined the cluster: check `GET /health/live` and cluster membership via `GET /cluster` for searcher nodes with the Searcher role enabled.","Start or restart searcher-capable nodes (ensure `searcher` is enabled in node config) and confirm they register in the searcher pool.","If searchers are running, check network connectivity / gossip ports between nodes so membership converges.","Add retry logic in the client path: searcher nodes leaving transiently (redeploy) is recoverable once they rejoin.","Check quickwit logs for searcher departures (graceful shutdown or failure) preceding the error."],"exampleFix":"// before: searching right after cluster start without checking readiness\nlet hits = client.search(request).await?;\n\n// after: wait until at least one searcher is available\nwait_until(|| {\n    let state = cluster_snapshot();\n    !state.searcher_nodes().is_empty()\n}).await?;\nlet hits = client.search(request).await?;","handlingStrategy":"retry","validationCode":"let searchers: Vec<_> = cluster.members_with_role(Role::Searcher);\nif searchers.is_empty() {\n    return Err(\"no searcher nodes available; retry after searchers join\");\n}","typeGuard":"fn has_searcher_nodes(pairs: &[(SocketAddr, SearcherNode)]) -> bool { !pairs.is_empty() }","tryCatchPattern":"match placer.assign_jobs(jobs, &excluded).await {\n    Ok(placement) => run_search(placement).await,\n    Err(e) if e.to_string().contains(\"no available searcher nodes\") => {\n        tokio::time::sleep(Duration::from_secs(2)).await;\n        retry_with_backoff().await\n    }\n    Err(e) => return Err(e),\n}","preventionTips":["Monitor cluster membership and alert when the searcher count drops to zero.","Deploy redundant searcher replicas so at least one is always available.","Gate query traffic on cluster readiness (at least one searcher live) at the load balancer.","Use graceful searcher shutdown/drain so pool churn is minimized during deploys."],"tags":["search","cluster","capacity","rust"],"backgroundTag":"empty-result-set","analyzedSha":"a39730c5cdcd1a4fe798403737ae293999ea21f8","analyzedAt":"2026-09-08T13:19:37.784Z","contentChangedAt":"2026-09-08T13:19:37.784Z","schemaVersion":2},"datasetVersion":"2026-09-14T11:17:12.474Z"}