apache/cassandra · critical · RuntimeException

Unable to stream hints since no live endpoints seen

Error message

Unable to stream hints since no live endpoints seen

What it means

When a node leaves the ring (e.g., decommission) it must hand off hinted handoff data to a live peer. If getStreamCandidates returns no live endpoints, there is nowhere to stream hints, so RuntimeException is thrown and the operation aborts rather than silently dropping hints.

Solutions

1. Ensure at least one peer node is UP before decommissioning (nodetool status)
2. On a single-node cluster, hints need no streaming — stop the node normally instead of decommissioning
3. Restart failed peers or fix network connectivity, then retry decommission

Example fix

// before
nodetool decommission   # on a 1-node cluster
// after
nodetool drain && systemctl stop cassandra   # single node: just stop it

if (livePeerCount() == 0)
    throw new IllegalStateException("No live peers; cannot stream hints during decommission");

try { ss.decommission(); } catch (RuntimeException e) { /* check nodetool status; bring peers up and retry */ }

When it happens

Trigger: Calling nodetool decommission (or drain/leave paths that stream hints) on a single-node cluster or when every other node in the datacenter/ring is down — no live candidates exist to receive the hints.

Common situations: Decommissioning the last node of a cluster; decommissioning while all peers are failed; network partition isolating the node during leave; testing decommission on a one-node dev cluster.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

• Hint dispatch was interrupted
• hints_directory must not be the same as any…
• a hints file cannot be configured for both compression and…
• A maximum number of tokens per node is supported
• A repair_session_space of

AI-assisted analysis of apache/cassandra@88fd0f6a0e (2026-09-10). Data as JSON: /api/errors/2b6b35dde18d6551. Report an issue: GitHub.

Appendix: source

Thrown at src/java/org/apache/cassandra/service/StorageService.java:3624

        return SystemReplicas.getSystemReplicas(endpoints);
    }
    /**
     * Find the best target to stream hints to. Currently the closest peer according to the snitch
     */
    private UUID getPreferredHintsStreamTarget()
    {
        ClusterMetadata metadata = ClusterMetadata.current();

        Set<InetAddressAndPort> endpoints = metadata.directory.states.entrySet().stream()
                                                                            .filter(e -> e.getValue() != NodeState.LEAVING)
                                                                            .map(e -> metadata.directory.endpoint(e.getKey()))
                                                                            .collect(toSet());

        EndpointsForRange candidates = getStreamCandidates(endpoints);
        if (candidates.isEmpty())
        {
            logger.warn("Unable to stream hints since no live endpoints seen");
            throw new RuntimeException("Unable to stream hints since no live endpoints seen");
        }
        else
        {
            // stream to the closest peer as chosen by the configured proximity measures
            candidates = DatabaseDescriptor.getNodeProximity().sortedByProximity(getBroadcastAddressAndPort(), candidates);
            InetAddressAndPort hintsDestinationHost = candidates.get(0).endpoint();
            return ClusterMetadata.current().directory.peerId(hintsDestinationHost).toUUID();
        }
    }

    public void move(String newToken)
    {
        try
        {
            getTokenFactory().validate(newToken);
        }
        catch (ConfigurationException e)
        {

View on GitHub (pinned to 88fd0f6a0e)