received an invalid gossip generation for peer

Error message

received an invalid gossip generation for peer {}; local time = {}, received generation = {}

What it means

During gossip digest comparison, if a peer advertises a generation timestamp more than MAX_GENERATION_DIFFERENCE seconds ahead of local time, Gossiper concludes the value is corrupt or fabricated and logs a warning instead of accepting it. Generations are wall-clock-based, so a future-dated generation would wrongly win every heartbeat comparison.

Solutions

Fix the clock on the offending peer (NTP/chrony) and restart Cassandra so the generation is re-initialized.
If a node was cloned/restored, ensure the clone's clock and generation are sane before rejoining.
Confirm no node's clock is more than MAX_GENERATION_DIFFERENCE ahead of the rest; keep NTP skew under a few hundred ms.
If data was corrupted, remove and re-bootstrap the offending node.

Example fix

// before: node with drifting clock, no sync
// after
timedatectl set-ntp true
chronyc tracking   # verify offset < 200ms
# then restart Cassandra

Defensive patterns

Strategy: validation

Validate before calling

long skew = Math.abs(System.currentTimeMillis()/1000 - ntpTimeSeconds()); if (skew > 300) throw new IllegalStateException("Clock skew too large: " + skew + "s");

Prevention

Run NTP/chrony on every node and alert on skew > ~200ms.
Never clone running Cassandra VMs; use clean restarts.
After restore, verify system clock before starting Cassandra.
Monitor generation values in nodetool gossipinfo for anomalies.

When it happens

Trigger: applyStateLocally receives a remote state whose generation exceeds localTime + MAX_GENERATION_DIFFERENCE for endpoint ep.

Common situations: Peer with a badly wrong system clock (hours/days ahead); corrupted commit-log/system tables after a restore; operator manually editing gossip state; clock drift after VM migration or RTC failure.

Related errors

AI-assisted analysis of apache/cassandra@88fd0f6a0e (2026-09-10). Data as JSON: /api/errors/b2285c0dc148bb0d. Report an issue: GitHub.

Appendix: source

Thrown at src/java/org/apache/cassandra/gms/Gossiper.java:1508

            remoteState.removeMajorVersion3LegacyApplicationStates();

            /*
                If state does not exist just add it. If it does then add it if the remote generation is greater.
                If there is a generation tie, attempt to break it by heartbeat version.
            */
            if (localEpStatePtr != null)
            {
                int localGeneration = localEpStatePtr.getHeartBeatState().getGeneration();
                int remoteGeneration = remoteState.getHeartBeatState().getGeneration();
                long localTime = currentTimeMillis() / 1000;
                if (logger.isTraceEnabled())
                    logger.trace("{} local generation {}, remote generation {}", ep, localGeneration, remoteGeneration);

                // We measure generation drift against local time, based on the fact that generation is initialized by time
                if (remoteGeneration > localTime + MAX_GENERATION_DIFFERENCE)
                {
                    // assume some peer has corrupted memory and is broadcasting an unbelievable generation about another peer (or itself)
                    logger.warn("received an invalid gossip generation for peer {}; local time = {}, received generation = {}", ep, localTime, remoteGeneration);
                }
                else if (remoteGeneration > localGeneration)
                {
                    if (logger.isTraceEnabled())
                        logger.trace("Updating heartbeat state generation to {} from {} for {}", remoteGeneration, localGeneration, ep);
                    // major state change will handle the update by inserting the remote state directly
                    handleMajorStateChange(ep, remoteState);
                }
                else if (remoteGeneration == localGeneration) // generation has not changed, apply new states
                {
                    /* find maximum state */
                    int localMaxVersion = getMaxEndpointStateVersion(localEpStatePtr);
                    int remoteMaxVersion = getMaxEndpointStateVersion(remoteState);
                    if (remoteMaxVersion > localMaxVersion)
                    {
                        // apply states, but do not notify since there is no major change
                        applyNewStates(ep, localEpStatePtr, remoteState);
                    }

View on GitHub (pinned to 88fd0f6a0e)