{"record":{"id":"86dc90218882efd1","repo":"elastic/elasticsearch","slug":"nodeselector-rejected-all-nodes-living-an","errorCode":null,"errorMessage":"NodeSelector [{}] rejected all nodes, living {} and dead {}","messagePattern":"NodeSelector \\[(.+?)\\] rejected all nodes, living (.+?) and dead (.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"client/rest/src/main/java/org/elasticsearch/client/RestClient.java","lineNumber":521,"sourceCode":"         * that the NodeSelectors are OK with. We do this by passing the dead\n         * nodes through the NodeSelector so it can have its say in which nodes\n         * are ok. If the selector is ok with any of the nodes then we will take\n         * the one in the list that has the lowest revival time and try it.\n         */\n        if (false == deadNodes.isEmpty()) {\n            final List<DeadNode> selectedDeadNodes = new ArrayList<>(deadNodes);\n            /*\n             * We'd like NodeSelectors to remove items directly from deadNodes\n             * so we can find the minimum after it is filtered without having\n             * to compare many things. This saves us a sort on the unfiltered\n             * list.\n             */\n            nodeSelector.select(() -> new DeadNodeIteratorAdapter(selectedDeadNodes.iterator()));\n            if (false == selectedDeadNodes.isEmpty()) {\n                return singletonList(Collections.min(selectedDeadNodes).node);\n            }\n        }\n        throw new IOException(\"NodeSelector [\" + nodeSelector + \"] rejected all nodes, living \" + livingNodes + \" and dead \" + deadNodes);\n    }\n\n    /**\n     * Called after each successful request call.\n     * Receives as an argument the host that was used for the successful request.\n     */\n    private void onResponse(Node node) {\n        DeadHostState removedHost = this.blacklist.remove(node.getHost());\n        if (logger.isDebugEnabled() && removedHost != null) {\n            logger.debug(\"removed [\" + node + \"] from blacklist\");\n        }\n    }\n\n    /**\n     * Called after each failed attempt.\n     * Receives as an argument the host that was used for the failed attempt.\n     */\n    private void onFailure(Node node) {","sourceCodeStart":503,"sourceCodeEnd":539,"githubUrl":"https://github.com/elastic/elasticsearch/blob/db6a809a667c081ca1dc7500389d26975573215f/client/rest/src/main/java/org/elasticsearch/client/RestClient.java#L503-L539","documentation":"Thrown by RestClient.selectNodes when the configured NodeSelector removes every candidate node from both the living set and the dead (blacklisted) set. The client cannot pick a host to send the request to, so it fails the call with an IOException rather than guessing. The message includes the selector that did the rejecting plus the living/dead node lists for diagnosis.","triggerScenarios":"A custom or built-in NodeSelector (e.g. one that filters nodes by role, attribute, or cloud metadata) returns an empty iterable for both living and dead nodes. This is hit on every performRequest once all nodes are blacklisted as dead AND the selector also rejects every dead node's revival candidate.","commonSituations":"NodeSelector configured to require an attribute (e.g. 'data_only', 'warm') that no running node has; all cluster nodes became dead due to connectivity failure and a non-default selector keeps rejecting the dead-revival fallback; mismatch between selector expectations and the actual nodes-info sniffed hosts.","solutions":["Inspect the message's living/dead lists: if dead is non-empty but rejected, relax or replace the NodeSelector so it permits at least one dead node for revival.","Verify the cluster actually has nodes matching whatever attribute/role your NodeSelector requires (GET _nodes/http).","If all nodes are dead, check connectivity / endpoints and let the blacklist revival window elapse, or rebuild the RestClient with correct host list.","Use RestClientBuilder's default selector (accepts any node) unless you genuinely need to restrict nodes."],"exampleFix":"// before\nrestClientBuilder.setNodeSelector(nodes -> {\n    for (Node n : nodes) if (n.getRoles().isDataEligible()) return; // removes everything else\n    // implicitly rejects all if none matched\n});\n// after\nrestClientBuilder.setNodeSelector(nodes -> {\n    // keep default behaviour: never reject everything; only prefer data nodes\n    for (Node n : nodes) if (!n.getRoles().isDataEligible()) nodes.remove(); // still keeps master-only as dead-revival fallback\n});","handlingStrategy":"try-catch","validationCode":"// before performing requests, sanity-check that the selector can accept at least one configured node\nList<Node> sample = client.getNodes(); // or sniffed list\nNodeSelector sel = yourSelector;\nList<Node> copy = new ArrayList<>(sample);\nsel.select(copy::iterator); // remove via iterator\nif (copy.isEmpty()) throw new IllegalStateException(\"NodeSelector rejects every configured node: \" + sample);","typeGuard":null,"tryCatchPattern":"try {\n    Response r = client.performRequest(req);\n} catch (IOException e) {\n    if (e.getMessage() != null && e.getMessage().startsWith(\"NodeSelector [\")) {\n        // selector rejected all nodes: log living/dead lists, fall back to a different client/selector\n        log.warn(\"all nodes rejected by selector\", e);\n        // optionally: rebuild client with relaxed selector or alert ops\n    } else throw e;\n}","preventionTips":["Never write a NodeSelector that can remove every node unconditionally; always keep a fallback branch.","Log the selector and node list at startup to confirm at least one node passes.","Prefer the default selector unless you have a hard role/attribute requirement."],"tags":["rest-client","node-selector","load-balancing","network"],"backgroundTag":null,"analyzedSha":"db6a809a667c081ca1dc7500389d26975573215f","analyzedAt":"2026-08-12T01:39:14.192Z","schemaVersion":2},"datasetVersion":"2026-08-12T13:17:24.610Z"}