apache/cassandra · warning

Invalid endpoint state for {}; {} - {}

Error message

Invalid endpoint state for {}; {} - {}

What it means

GossipHelper.isValidForClusterMetadata checks that every gossip EndpointState carries all ApplicationState values required to build ClusterMetadata. If any required state is missing for a node, it logs a warning naming the endpoint, the missing state, and the full epstates map, and returns false so the gossip-to-TCM upgrade path refuses to proceed.

Source

Thrown at src/java/org/apache/cassandra/tcm/compatibility/GossipHelper.java:464

                                   InProgressSequences.EMPTY,
                                   ConsensusMigrationState.EMPTY,
                                   extensions,
                                   AccordStaleReplicas.EMPTY,
                                   CMSMembership.EMPTY);
    }

    public static boolean isValidForClusterMetadata(Map<InetAddressAndPort, EndpointState> epstates)
    {
        if (epstates.isEmpty())
            return false;
        EnumSet<ApplicationState> requiredStates = EnumSet.of(DC, RACK, HOST_ID, RELEASE_VERSION);
        for (Map.Entry<InetAddressAndPort, EndpointState> entry : epstates.entrySet())
        {
            EndpointState epstate = entry.getValue();
            for (ApplicationState state : requiredStates)
                if (epstate.getApplicationState(state) == null)
                {
                    logger.warn("Invalid endpoint state for {}; {} - {}", entry.getKey(), state, epstates);
                    return false;
                }
        }
        return true;
    }

    private static boolean containsDuplicateHostIds(Map<InetAddressAndPort, EndpointState> epstates)
    {
        Set<String> hostIds = new HashSet<>();
        for (EndpointState epstate : epstates.values())
        {
            VersionedValue vv = epstate.getApplicationState(HOST_ID);
            if (vv == null)
                continue;
            String hostIdString = vv.value;
            if (hostIds.contains(hostIdString))
                return true;
            hostIds.add(hostIdString);

View on GitHub (pinned to 88fd0f6a0e)

Solutions

  1. Ensure every node in the ring is running a TCM-capable version and fully joined (check gossip state via nodetool gossipinfo).
  2. Restart or repair the offending node so it republishes its ApplicationState (host ID, tokens, DC/RACK).
  3. Wait for gossip to converge and retry the upgrade/compatibility step.
  4. Remove decommissioned/ghost endpoints that permanently lack required states.

Example fix

// before: proceeding to convert gossip on incomplete state
if (GossipHelper.isValidForClusterMetadata(epstates)) { convert(); }
// after: gate on full membership readiness
if (liveNodes.stream().allMatch(this::hasRequiredStates) && GossipHelper.isValidForClusterMetadata(epstates)) { convert(); }
Defensive patterns

Strategy: validation

Validate before calling

// Check required gossip states on all endpoints before TCM conversion
Set<ApplicationState> required = EnumSet.of(ApplicationState.HOST_ID, ApplicationState.TOKENS, ApplicationState.DC);
boolean ready = epstates.values().stream()
    .allMatch(es -> required.stream().allMatch(s -> es.getApplicationState(s) != null));

Try / catch

// Wait for convergence instead of failing fast
if (!GossipHelper.isValidForClusterMetadata(epstates)) {
    Uninterruptibles.sleepUninterruptibly(10, TimeUnit.SECONDS);
    // re-fetch epstates and retry
}

Prevention

When it happens

Trigger: During upgrade from gossip-based metadata to TCM (GossipHelper.fromEndpointStates path), when a node in the gossip ring has not yet published one of the required ApplicationState entries (e.g. host ID, tokens, or schema version) because it just joined, is partially up, or is running an older version.

Common situations: Rolling upgrades where some nodes still run pre-TCM gossip semantics; a node crashed mid-join leaving incomplete endpoint state; racing the upgrade before all nodes published required states.

Understand the failure class

Background: "is required", "must be set", "missing required field": configuration validation errors across open-source libraries — this error's family across 36 libraries.

Related errors

AI-assisted analysis of apache/cassandra@88fd0f6a0e (2026-09-10). Data as JSON: /api/errors/ec2b2d69bea77de2. Report an issue: GitHub.