elastic/elasticsearch · error · IOException

NodeSelector [{}] rejected all nodes, living {} and dead {}

Error message

NodeSelector [{}] rejected all nodes, living {} and dead {}

What it means

Thrown by RestClient.selectNodes when the configured NodeSelector removes every candidate node from both the living set and the dead (blacklisted) set. The client cannot pick a host to send the request to, so it fails the call with an IOException rather than guessing. The message includes the selector that did the rejecting plus the living/dead node lists for diagnosis.

Source

Thrown at client/rest/src/main/java/org/elasticsearch/client/RestClient.java:521

         * that the NodeSelectors are OK with. We do this by passing the dead
         * nodes through the NodeSelector so it can have its say in which nodes
         * are ok. If the selector is ok with any of the nodes then we will take
         * the one in the list that has the lowest revival time and try it.
         */
        if (false == deadNodes.isEmpty()) {
            final List<DeadNode> selectedDeadNodes = new ArrayList<>(deadNodes);
            /*
             * We'd like NodeSelectors to remove items directly from deadNodes
             * so we can find the minimum after it is filtered without having
             * to compare many things. This saves us a sort on the unfiltered
             * list.
             */
            nodeSelector.select(() -> new DeadNodeIteratorAdapter(selectedDeadNodes.iterator()));
            if (false == selectedDeadNodes.isEmpty()) {
                return singletonList(Collections.min(selectedDeadNodes).node);
            }
        }
        throw new IOException("NodeSelector [" + nodeSelector + "] rejected all nodes, living " + livingNodes + " and dead " + deadNodes);
    }

    /**
     * Called after each successful request call.
     * Receives as an argument the host that was used for the successful request.
     */
    private void onResponse(Node node) {
        DeadHostState removedHost = this.blacklist.remove(node.getHost());
        if (logger.isDebugEnabled() && removedHost != null) {
            logger.debug("removed [" + node + "] from blacklist");
        }
    }

    /**
     * Called after each failed attempt.
     * Receives as an argument the host that was used for the failed attempt.
     */
    private void onFailure(Node node) {

View on GitHub (pinned to db6a809a66)

Solutions

1. Inspect the message's living/dead lists: if dead is non-empty but rejected, relax or replace the NodeSelector so it permits at least one dead node for revival.
2. Verify the cluster actually has nodes matching whatever attribute/role your NodeSelector requires (GET _nodes/http).
3. If all nodes are dead, check connectivity / endpoints and let the blacklist revival window elapse, or rebuild the RestClient with correct host list.
4. Use RestClientBuilder's default selector (accepts any node) unless you genuinely need to restrict nodes.

Example fix

// before
restClientBuilder.setNodeSelector(nodes -> {
    for (Node n : nodes) if (n.getRoles().isDataEligible()) return; // removes everything else
    // implicitly rejects all if none matched
});
// after
restClientBuilder.setNodeSelector(nodes -> {
    // keep default behaviour: never reject everything; only prefer data nodes
    for (Node n : nodes) if (!n.getRoles().isDataEligible()) nodes.remove(); // still keeps master-only as dead-revival fallback
});

// before performing requests, sanity-check that the selector can accept at least one configured node
List<Node> sample = client.getNodes(); // or sniffed list
NodeSelector sel = yourSelector;
List<Node> copy = new ArrayList<>(sample);
sel.select(copy::iterator); // remove via iterator
if (copy.isEmpty()) throw new IllegalStateException("NodeSelector rejects every configured node: " + sample);

try {
    Response r = client.performRequest(req);
} catch (IOException e) {
    if (e.getMessage() != null && e.getMessage().startsWith("NodeSelector [")) {
        // selector rejected all nodes: log living/dead lists, fall back to a different client/selector
        log.warn("all nodes rejected by selector", e);
        // optionally: rebuild client with relaxed selector or alert ops
    } else throw e;
}

When it happens

Trigger: A custom or built-in NodeSelector (e.g. one that filters nodes by role, attribute, or cloud metadata) returns an empty iterable for both living and dead nodes. This is hit on every performRequest once all nodes are blacklisted as dead AND the selector also rejects every dead node's revival candidate.

Common situations: NodeSelector configured to require an attribute (e.g. 'data_only', 'warm') that no running node has; all cluster nodes became dead due to connectivity failure and a non-default selector keeps rejecting the dead-revival fallback; mismatch between selector expectations and the actual nodes-info sniffed hosts.

Related errors

• IO error while waiting cluster
• Failed to pull Docker base image [{baseImage}], all attempts
• Gradle is running in offline mode, but branches.json locatio
• Failed to download branches.json from: {}
• Failed to fetch branch {normalizedBranch} from {sourceRepo}

AI-assisted analysis of elastic/elasticsearch@db6a809a66 (2026-08-12). Data as JSON: /api/errors/86dc90218882efd1. Report an issue: GitHub.