prestodb/presto · error · PrestoException

NO_NODES_AVAILABLE

NO_NODES_AVAILABLE

Error message

No nodes available to run query

What it means

TopologyAwareNodeSelector.computeAssignments throws NO_NODES_AVAILABLE when a HARD_AFFINITY split's preferred nodes, resolved through topology-aware node selection, yield an empty candidate set - i.e. none of the split's preferred (e.g. HDFS datanode/locality-preferred) nodes are active in the cluster. The topology-aware scheduler maps split host preferences through network topology; if the mapped nodes are all down or excluded (including the coordinator being excluded), the split cannot be placed and the query fails fast.

Source

Thrown at presto-main-base/src/main/java/com/facebook/presto/execution/scheduler/nodeSelection/TopologyAwareNodeSelector.java:161

    public SplitPlacementResult computeAssignments(Set<Split> splits, List<RemoteTask> existingTasks)
    {
        NodeMap nodeMap = this.nodeMap.get().get();
        Multimap<InternalNode, Split> assignment = HashMultimap.create();
        NodeAssignmentStats assignmentStats = new NodeAssignmentStats(nodeTaskMap, nodeMap, existingTasks);

        int[] topologicCounters = new int[topologicalSplitCounters.size()];
        Set<NetworkLocation> filledLocations = new HashSet<>();
        Set<InternalNode> blockedExactNodes = new HashSet<>();
        boolean splitWaitingForAnyNode = false;

        NodeProvider nodeProvider = nodeMap.getNodeProvider(maxPreferredNodes);
        for (Split split : splits) {
            SplitWeight splitWeight = split.getSplitWeight();
            if (split.getNodeSelectionStrategy() == HARD_AFFINITY) {
                List<InternalNode> candidateNodes = selectExactNodes(nodeMap, split.getPreferredNodes(nodeProvider), includeCoordinator);
                if (candidateNodes.isEmpty()) {
                    log.debug("No nodes available to schedule %s. Available nodes %s", split, nodeMap.getActiveNodes());
                    throw new PrestoException(NO_NODES_AVAILABLE, "No nodes available to run query");
                }
                InternalNode chosenNode = bestNodeSplitCount(splitWeight, candidateNodes.iterator(), minCandidates, maxPendingSplitsWeightPerTask, assignmentStats);
                if (chosenNode != null) {
                    assignment.put(chosenNode, split);
                    assignmentStats.addAssignedSplit(chosenNode, splitWeight);
                }
                // Exact node set won't matter, if a split is waiting for any node
                else if (!splitWaitingForAnyNode) {
                    blockedExactNodes.addAll(candidateNodes);
                }
                continue;
            }

            InternalNode chosenNode = null;
            int depth = networkLocationSegmentNames.size();
            int chosenDepth = 0;
            Set<NetworkLocation> locations = new HashSet<>();
            for (HostAddress host : split.getPreferredNodes(nodeProvider)) {

View on GitHub (pinned to 55bb57d202)

Solutions

1. Bring the preferred worker nodes back online or restore the connector's underlying storage nodes (e.g. HDFS datanodes).
2. Set node-scheduler.include-coordinator=true to widen the eligible node set.
3. Check topology configuration (network-topology segments/file) so split hosts map to real cluster nodes.
4. Disable bucketed/locality-affinity execution for the affected connector (e.g. hive.bucketed-execution=false) or rewrite data to distribute blocks across live nodes.
5. Inspect coordinator logs 'No nodes available to schedule <split>' to identify which preferred nodes were missing.

Example fix

// before (etc/config.properties)
node-scheduler.include-coordinator=false
// HDFS locality-preferred nodes all down -> NO_NODES_AVAILABLE

// after
node-scheduler.include-coordinator=true
// plus: repair/replace dead HDFS datanodes hosting the splits

// Verify the split's preferred nodes are alive before topology-aware assignment
List<InternalNode> preferred = selectExactNodes(nodeMap, split.getPreferredNodes(nodeProvider), includeCoordinator);
if (preferred.isEmpty()) {
    throw new IllegalStateException("Preferred nodes for split are not active: " + split);
}

private static boolean hasLivePreferredNodes(Split split, NodeMap nodeMap, NodeProvider provider, boolean includeCoordinator) {
    return !selectExactNodes(nodeMap, split.getPreferredNodes(provider), includeCoordinator).isEmpty();
}

try {
    return topologyAwareNodeSelector.computeAssignments(splits, existingTasks);
} catch (PrestoException e) {
    if (e.getErrorCode() == StandardErrorCode.NO_NODES_AVAILABLE.toErrorCode().getCode()) {
        // degrade to simple node selection or wait for preferred nodes to rejoin
        return simpleNodeSelector.computeAssignments(splits, existingTasks);
    }
    throw e;
}

When it happens

Trigger: A split with HARD_AFFINITY (bucketed execution, connector-preferred localities like HDFS block locations) whose preferredNodes after topology resolution via selectExactNodes yield no active nodes; preferred nodes decommissioned or crashed; include-coordinator=false and only the coordinator matches the locality.

Common situations: HDFS datanodes hosting the data blocks are down while running topology-aware scheduling with bucketed execution; rack/node topology configuration (node-scheduler.network-topology settings) misconfigured so locality resolves to nonexistent hosts; decommissioning workers that host hot data; autoscaling shrinks the cluster below the split's locality requirements.

Related errors

• NO_NODES_AVAILABLE
• NO_NODES_AVAILABLE
• NODE_SELECTION_NOT_SUPPORTED
• INVALID_TABLE_PROPERTY
• ACCUMULO_TABLE_EXISTS

AI-assisted analysis of prestodb/presto@55bb57d202 (2026-09-04). Data as JSON: /api/errors/39ed1171105bd0d4. Report an issue: GitHub.