apache/cassandra · error · IllegalStateException

Unknown endpoint in at

Error message

Unknown endpoint %s in %s at %s

What it means

ReplicaGroups.byNodeId maps each endpoint of every range group to a NodeId via the supplied Directory.IdLookup. It throws IllegalStateException when an endpoint present in the placement data has no known NodeId in the directory, meaning placements reference an endpoint the directory does not know about.

Solutions

1. Repair cluster metadata so placements and directory are consistent (recompute placements after the node is fully registered or removed)
2. Ensure the node's directory entry exists before running placement transformations referencing its endpoint
3. Roll back / reapply the in-progress metadata transformation (e.g. restart the CMS operation) so intermediate states are not persisted
4. Audit Directory vs PlacementDeltas for the failing endpoint and remove orphaned replicas

Example fix

// before
NodeId nodeId = idLookup.peerId(endpoint);
// after
NodeId nodeId = idLookup.peerId(endpoint);
if (nodeId == null)
    logger.warn("Skipping unknown endpoint {} not present in directory", endpoint);
else
    builder.put(nodeId, new ReplicaNode(nodeId, ...));

for (InetAddressAndPort ep : endpointsInPlacements)
    if (metadata.directory.peerId(ep) == null)
        throw new IllegalStateException("endpoint missing from directory: " + ep);

try { Map<NodeId, ReplicaNode> m = replicaGroups.byNodeId(idLookup); } catch (IllegalStateException e) { /* rebuild placements from fresh metadata */ }

When it happens

Trigger: Calling byNodeId (used by oldMap/newMap during placement computations) while the cluster metadata directory lacks an entry for an endpoint that still appears in placement ranges — e.g. after a node was removed from the directory but its replicas remain in placements.

Common situations: Interrupted decommission leaves stale replicas in placement metadata; a join failed after range assignment but before directory registration was completed; metadata snapshot inconsistency between placements and directory epochs.

Understand the failure class

Background: "Not found" and "does not exist" errors: why "Task not found", "No such folder", and "Can't find" fire when a lookup comes back empty — this error's family across 14 libraries.

Related errors

• Can only initialize cluster identifier during epoch
• Can't revert join from
• Can't revert replacement from
• Could not find range for token in ReplicaGroups:
• Failed to find first CMS node in directory

AI-assisted analysis of apache/cassandra@88fd0f6a0e (2026-09-10). Data as JSON: /api/errors/5c0f369cf1164348. Report an issue: GitHub.

Appendix: source

Thrown at src/java/org/apache/cassandra/tcm/ownership/ReplicaGroups.java:188

    public Delta difference(NodeIdLookup idLookup, ReplicaGroups next)
    {
        Multimap<NodeId, ReplicaNode> oldMap = this.byNodeId(idLookup);
        Multimap<NodeId, ReplicaNode> newMap = next.byNodeId(idLookup);
        return new NodeIdDelta(diff(oldMap, newMap), diff(newMap, oldMap));
    }

    private Multimap<NodeId, ReplicaNode> byNodeId(NodeIdLookup idLookup)
    {
        ImmutableMultimap.Builder<NodeId, ReplicaNode> builder = ImmutableMultimap.builder();
        for (int i = 0; i < endpoints.size(); i++)
        {
            Map<InetAddressAndPort, Replica> replica = endpoints.get(i).byEndpoint();
            for (Map.Entry<InetAddressAndPort, Replica> entry : replica.entrySet())
            {
                InetAddressAndPort endpoint = entry.getKey();
                NodeId nodeId = idLookup.peerId(endpoint);
                if (nodeId == null)
                    throw new IllegalStateException(String.format("Unknown endpoint %s in %s at %s", endpoint, idLookup, idLookup.lastModified()));
                builder.put(nodeId, new ReplicaNode(nodeId, entry.getValue().range(), entry.getValue().isFull()));
            }
        }
        return builder.build();
    }

    @VisibleForTesting
    public RangesByEndpoint byEndpoint()
    {
        RangesByEndpoint.Builder builder = new RangesByEndpoint.Builder();
        for (int i = 0; i < endpoints.size(); i++)
            endpoints.get(i).byEndpoint().forEach(builder::put);
        return builder.build();
    }

    private static ImmutableMultimap<NodeId, ReplicaNode> diff(Multimap<NodeId, ReplicaNode> left, Multimap<NodeId, ReplicaNode> right)
    {
        ImmutableMultimap.Builder<NodeId, ReplicaNode> builder = ImmutableMultimap.builder();

View on GitHub (pinned to 88fd0f6a0e)