prestodb/presto · critical · ResourceManagerInconsistentException

%s nodes found in discovery vs. %s nodes found in heartbeats

Error message

%s nodes found in discovery vs. %s nodes found in heartbeats

What it means

ResourceManagerClusterStateProvider validates that every coordinator known to discovery is also seen in the node-status heartbeats the resource manager receives. If the discovered coordinator set and the heartbeated coordinator set differ (or there are no coordinators at all), the cluster view is inconsistent, so getClusterResourceGroups throws ResourceManagerInconsistentException instead of returning stale resource-group state.

Source

Thrown at presto-main-base/src/main/java/com/facebook/presto/resourcemanager/ResourceManagerClusterStateProvider.java:373

    public Map<String, MemoryInfo> getWorkerMemoryInfo()
    {
        return nodeStatuses.entrySet().stream().collect(toImmutableMap(e -> {
            String nodeIdentifier = e.getValue().getNodeStatus().getNodeId();
            String nodeHost = URI.create(e.getValue().getNodeStatus().getExternalAddress()).getHost();
            return nodeIdentifier + " [" + nodeHost + "]";
        }, e -> e.getValue().getNodeStatus().getMemoryInfo()));
    }

    private void validateCoordinatorConsistency()
    {
        Set<String> coordinators = internalNodeManager.getCoordinators().stream().map(InternalNode::getNodeIdentifier).collect(toImmutableSet());
        Set<String> heartbeatedCoordinatorNodes = nodeStatuses.values().stream()
                .map(InternalNodeState::getNodeStatus)
                .filter(NodeStatus::isCoordinator)
                .map(NodeStatus::getNodeId)
                .collect(toImmutableSet());
        if (!(Sets.difference(coordinators, heartbeatedCoordinatorNodes).isEmpty() && !coordinators.isEmpty())) {
            throw new ResourceManagerInconsistentException(format("%s nodes found in discovery vs. %s nodes found in heartbeats", coordinators.size(), heartbeatedCoordinatorNodes.size()));
        }
    }

    private static class CoordinatorResourceGroupState
    {
        private final String nodeId;
        private final long lastHeartbeatInMillis;
        private final List<ResourceGroupRuntimeInfo> resourceGroups;

        public CoordinatorResourceGroupState(
                String nodeId,
                long lastHeartbeatInMillis,
                List<ResourceGroupRuntimeInfo> resourceGroups)
        {
            this.nodeId = requireNonNull(nodeId, "nodeId is null");
            this.lastHeartbeatInMillis = lastHeartbeatInMillis;
            this.resourceGroups = requireNonNull(resourceGroups, "resourceGroups is null");
        }

View on GitHub (pinned to 55bb57d202)

Solutions

Check every coordinator process is up and heartbeating to the resource manager (logs on both sides)
Verify network connectivity between each coordinator and the resource manager (ports, firewalls, DNS)
Confirm all coordinators share the same discovery URI / resource-manager configuration
If transient (startup or rolling restart), retry after heartbeats converge
If a coordinator was removed, wait for its discovery entry to expire or clean stale discovery state

Defensive patterns

Strategy: try-catch

Validate before calling

// gauge cluster health before scheduling
boolean consistent = discoveredCoordinators.equals(heartbeatedCoordinators) && !discoveredCoordinators.isEmpty();

Try / catch

try {
    provider.getClusterResourceGroups();
} catch (ResourceManagerInconsistentException e) {
    log.warn("Discovery/heartbeat coordinator mismatch: {}", e.getMessage());
    // retry after heartbeat interval or report unhealthy cluster
}

Prevention

Monitor heartbeat lags between coordinators and resource manager
Alert when discovery node count diverges from heartbeat node count
Use identical discovery configuration on all coordinators
Retry transient mismatches during startup instead of failing hard

When it happens

Trigger: validateCoordinatorConsistency, called from getClusterResourceGroups, throws when Sets.difference(coordinators, heartbeatedCoordinatorNodes) is non-empty or when the discovered coordinators set is empty — i.e. a coordinator appears in discovery but its NodeStatus heartbeats have not reached this resource manager.

Common situations: A coordinator crashed or was killed but its discovery entry has not expired; network partition between a coordinator and the resource manager blocking heartbeats; rolling restart/redeployment where discovery and heartbeat state are momentarily out of sync; misconfigured discovery URI so a coordinator registers in discovery but heartbeats elsewhere; very fresh cluster where heartbeats have not all arrived yet.

Related errors

AI-assisted analysis of prestodb/presto@55bb57d202 (2026-09-04). Data as JSON: /api/errors/4107d9508c533426. Report an issue: GitHub.