{"record":{"id":"b06ee2ae71cf8de3","repo":"quickwit-oss/quickwit","slug":"failed-to-assign-search-jobs-there-are-no-searche","errorCode":null,"errorMessage":"failed to assign search jobs: there are no searcher nodes candidates for these jobs","messagePattern":"failed to assign search jobs: there are no searcher nodes candidates for these jobs","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"quickwit/quickwit-search/src/search_job_placer.rs","lineNumber":212,"sourceCode":"        &self,\n        mut jobs: Vec<J>,\n        excluded_addrs: &HashSet<SocketAddr>,\n        load_aware: bool,\n    ) -> anyhow::Result<impl Iterator<Item = (SearchServiceClient, Vec<J>)> + use<J>> {\n        let mut all_nodes = self.searcher_pool.pairs();\n\n        if all_nodes.is_empty() {\n            bail!(\n                \"failed to assign search jobs: there are no available searcher nodes in the \\\n                 cluster\"\n            );\n        }\n        if !excluded_addrs.is_empty() && excluded_addrs.len() < all_nodes.len() {\n            all_nodes.retain(|(grpc_addr, _)| !excluded_addrs.contains(grpc_addr));\n\n            // This should never happen, but... belt and suspenders policy.\n            if all_nodes.is_empty() {\n                bail!(\n                    \"failed to assign search jobs: there are no searcher nodes candidates for \\\n                     these jobs\"\n                );\n            }\n            info!(\n                \"excluded {} nodes from search job placement, {} remaining\",\n                excluded_addrs.len(),\n                all_nodes.len()\n            );\n        }\n        let mut candidate_nodes: Vec<CandidateNode> = all_nodes\n            .into_iter()\n            .map(|(grpc_addr, searcher_node)| CandidateNode {\n                affinity_id: searcher_node.node_id,\n                grpc_addr,\n                client: searcher_node.client,\n                load: None,\n            })","sourceCodeStart":194,"sourceCodeEnd":230,"githubUrl":"https://github.com/quickwit-oss/quickwit/blob/a39730c5cdcd1a4fe798403737ae293999ea21f8/quickwit/quickwit-search/src/search_job_placer.rs#L194-L230","documentation":"In `assign_jobs_inner`, after removing all nodes listed in `excluded_addrs` from the candidate pool, the retained list is empty. This is marked in the code as a belt-and-suspenders branch: exclusion only filters when `excluded_addrs.len() < all_nodes.len()`, so filtering to empty 'should never happen'. Hitting it means the node set changed between the size check and the `retain`, or the exclusion precondition was violated.","triggerScenarios":"Calling `assign_jobs()` / `assign_jobs_ignoring_load()` where the excluded-address filtering empties the candidate list — e.g., a concurrent cluster membership change removed nodes between `excluded_addrs.len() < all_nodes.len()` being checked and `all_nodes.retain(...)` executing, causing an internal invariant violation.","commonSituations":"High node churn (searchers flapping in and out of the Chitchat cluster) while searches carrying exclusion lists (e.g., nodes that previously failed a request) are being placed; race between cluster state refresh and job placement.","solutions":["Retry the placement: this is a race condition; a subsequent call will typically see a consistent cluster snapshot.","Reduce how aggressively failed nodes are excluded so exclusions never cover the whole pool, or cap exclusion count to `all_nodes.len() - 1`.","Investigate searcher flapping: check gossip connectivity and searcher crash logs causing rapid membership churn.","If reproducible, treat as a bug in the placer's consistency check and report/fix the check-rethen-retain race (re-check emptiness against the fresh list)."],"exampleFix":"// before: check size, then filter (race window)\nif !excluded_addrs.is_empty() && excluded_addrs.len() < all_nodes.len() {\n    all_nodes.retain(|(grpc_addr, _)| !excluded_addrs.contains(grpc_addr));\n}\n\n// after: filter first, fall back to unfiltered pool if emptied\nif !excluded_addrs.is_empty() {\n    let filtered: Vec<_> = all_nodes.into_iter()\n        .filter(|(addr, _)| !excluded_addrs.contains(addr))\n        .collect();\n    if !filtered.is_empty() { all_nodes = filtered; }\n}","handlingStrategy":"retry","validationCode":"// Prefer bounding exclusions at the caller:\nlet excluded: HashSet<SocketAddr> = failed_nodes\n    .into_iter()\n    .take(cluster.searcher_count().saturating_sub(1))\n    .collect();","typeGuard":null,"tryCatchPattern":"for attempt in 0..3 {\n    match placer.assign_jobs(jobs.clone(), &excluded).await {\n        Ok(p) => return run_search(p).await,\n        Err(e) if e.to_string().contains(\"no searcher nodes candidates\") => {\n            sleep(backoff(attempt)).await;\n            excluded.clear(); // fall back to full pool on next try\n        }\n        Err(e) => return Err(e),\n    }\n}","preventionTips":["Keep exclusion lists bounded so they can never cover the whole pool.","Retry placement with cleared exclusions on this specific error.","Investigate and stabilize node membership churn (gossip connectivity, crashes).","Treat this branch as an invariant: log it loudly if it reproduces, since the code declares it unreachable."],"tags":["search","cluster","race-condition","rust"],"backgroundTag":"internal-invariant-violation","analyzedSha":"a39730c5cdcd1a4fe798403737ae293999ea21f8","analyzedAt":"2026-09-08T13:19:37.784Z","contentChangedAt":"2026-09-08T13:19:37.784Z","schemaVersion":2},"datasetVersion":"2026-09-14T11:17:12.474Z"}