{"record":{"id":"b2285c0dc148bb0d","repo":"apache/cassandra","slug":"received-an-invalid-gossip-generation-for-peer","errorCode":null,"errorMessage":"received an invalid gossip generation for peer {}; local time = {}, received generation = {}","messagePattern":"received an invalid gossip generation for peer (.+?); local time = (.+?), received generation = (.+?)","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"src/java/org/apache/cassandra/gms/Gossiper.java","lineNumber":1508,"sourceCode":"            remoteState.removeMajorVersion3LegacyApplicationStates();\n\n            /*\n                If state does not exist just add it. If it does then add it if the remote generation is greater.\n                If there is a generation tie, attempt to break it by heartbeat version.\n            */\n            if (localEpStatePtr != null)\n            {\n                int localGeneration = localEpStatePtr.getHeartBeatState().getGeneration();\n                int remoteGeneration = remoteState.getHeartBeatState().getGeneration();\n                long localTime = currentTimeMillis() / 1000;\n                if (logger.isTraceEnabled())\n                    logger.trace(\"{} local generation {}, remote generation {}\", ep, localGeneration, remoteGeneration);\n\n                // We measure generation drift against local time, based on the fact that generation is initialized by time\n                if (remoteGeneration > localTime + MAX_GENERATION_DIFFERENCE)\n                {\n                    // assume some peer has corrupted memory and is broadcasting an unbelievable generation about another peer (or itself)\n                    logger.warn(\"received an invalid gossip generation for peer {}; local time = {}, received generation = {}\", ep, localTime, remoteGeneration);\n                }\n                else if (remoteGeneration > localGeneration)\n                {\n                    if (logger.isTraceEnabled())\n                        logger.trace(\"Updating heartbeat state generation to {} from {} for {}\", remoteGeneration, localGeneration, ep);\n                    // major state change will handle the update by inserting the remote state directly\n                    handleMajorStateChange(ep, remoteState);\n                }\n                else if (remoteGeneration == localGeneration) // generation has not changed, apply new states\n                {\n                    /* find maximum state */\n                    int localMaxVersion = getMaxEndpointStateVersion(localEpStatePtr);\n                    int remoteMaxVersion = getMaxEndpointStateVersion(remoteState);\n                    if (remoteMaxVersion > localMaxVersion)\n                    {\n                        // apply states, but do not notify since there is no major change\n                        applyNewStates(ep, localEpStatePtr, remoteState);\n                    }","sourceCodeStart":1490,"sourceCodeEnd":1526,"githubUrl":"https://github.com/apache/cassandra/blob/88fd0f6a0eaed8943f05ac9e8f947882b8ddc8f1/src/java/org/apache/cassandra/gms/Gossiper.java#L1490-L1526","documentation":"During gossip digest comparison, if a peer advertises a generation timestamp more than MAX_GENERATION_DIFFERENCE seconds ahead of local time, Gossiper concludes the value is corrupt or fabricated and logs a warning instead of accepting it. Generations are wall-clock-based, so a future-dated generation would wrongly win every heartbeat comparison.","triggerScenarios":"applyStateLocally receives a remote state whose generation exceeds localTime + MAX_GENERATION_DIFFERENCE for endpoint ep.","commonSituations":"Peer with a badly wrong system clock (hours/days ahead); corrupted commit-log/system tables after a restore; operator manually editing gossip state; clock drift after VM migration or RTC failure.","solutions":["Fix the clock on the offending peer (NTP/chrony) and restart Cassandra so the generation is re-initialized.","If a node was cloned/restored, ensure the clone's clock and generation are sane before rejoining.","Confirm no node's clock is more than MAX_GENERATION_DIFFERENCE ahead of the rest; keep NTP skew under a few hundred ms.","If data was corrupted, remove and re-bootstrap the offending node."],"exampleFix":"// before: node with drifting clock, no sync\n// after\ntimedatectl set-ntp true\nchronyc tracking   # verify offset < 200ms\n# then restart Cassandra","handlingStrategy":"validation","validationCode":"long skew = Math.abs(System.currentTimeMillis()/1000 - ntpTimeSeconds()); if (skew > 300) throw new IllegalStateException(\"Clock skew too large: \" + skew + \"s\");","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Run NTP/chrony on every node and alert on skew > ~200ms.","Never clone running Cassandra VMs; use clean restarts.","After restore, verify system clock before starting Cassandra.","Monitor generation values in nodetool gossipinfo for anomalies."],"tags":["cassandra","gossip","clock-skew","time-drift"],"backgroundTag":"clock-skew","analyzedSha":"88fd0f6a0eaed8943f05ac9e8f947882b8ddc8f1","analyzedAt":"2026-09-10T07:29:22.284Z","contentChangedAt":"2026-09-10T07:29:22.284Z","schemaVersion":2},"datasetVersion":"2026-09-17T15:17:12.973Z"}