{"record":{"id":"cbb87be496a8e7b7","repo":"apache/cassandra","slug":"err-wrong-machine-state","errorCode":"ERR_WRONG_MACHINE_STATE","errorMessage":"There are tables for which gc_grace_seconds is older than the lastly known time Cassandra node was up based on its heartbeat %s with timestamp %s. Cassandra node will not start as it would likely introduce data consistency issues (zombies etc). Please resolve these issues manually, then remove the heartbeat and start the node again. Invalid tables: %s","messagePattern":"There are tables for which gc_grace_seconds is older than the lastly known time Cassandra node was up based on its heartbeat (.+?) with timestamp (.+?)\\. Cassandra node will not start as it would likely introduce data consistency issues \\(zombies etc\\)\\. Please resolve these issues manually, then remove the heartbeat and start the node again\\. Invalid tables: (.+?)","errorType":"error_code","errorClass":"StartupException","httpStatus":null,"severity":"critical","filePath":"src/java/org/apache/cassandra/service/DataResurrectionCheck.java","lineNumber":286,"sourceCode":"                    violations.add(Pair.create(keyspace, userTable.table));\n            }\n        }\n\n        if (!violations.isEmpty())\n        {\n            String invalidTables = violations.stream()\n                                             .map(p -> format(\"%s.%s\", p.left, p.right))\n                                             .collect(joining(\",\"));\n\n            String exceptionMessage = format(\"There are tables for which gc_grace_seconds is older \" +\n                                             \"than the lastly known time Cassandra node was up based \" +\n                                             \"on its heartbeat %s with timestamp %s. Cassandra node will not start \" +\n                                             \"as it would likely introduce data consistency \" +\n                                             \"issues (zombies etc). Please resolve these issues manually, \" +\n                                             \"then remove the heartbeat and start the node again. Invalid tables: %s\",\n                                             heartbeatFile, heartbeat.lastHeartbeat, invalidTables);\n\n            throw new StartupException(ERR_WRONG_MACHINE_STATE, exceptionMessage);\n        }\n    }\n\n    @Override\n    public void postAction(StartupChecksConfiguration configuration)\n    {\n        // Schedule heartbeating after all checks have passed, not as part of the check,\n        // as it might happen that other checks after it might fail, but we would be heartbeating already.\n        if (!configuration.isEnabled(name()))\n            return;\n\n        Map<String, Object> configMap = configuration.getConfig(name());\n        File heartbeatFile = DataResurrectionCheck.getHeartbeatFile(configMap);\n\n        ScheduledExecutors.scheduledTasks.scheduleAtFixedRate(() ->\n        {\n            Heartbeat heartbeat = new Heartbeat(Instant.ofEpochMilli(Clock.Global.currentTimeMillis()));\n            try","sourceCodeStart":268,"sourceCodeEnd":304,"githubUrl":"https://github.com/apache/cassandra/blob/88fd0f6a0eaed8943f05ac9e8f947882b8ddc8f1/src/java/org/apache/cassandra/service/DataResurrectionCheck.java#L268-L304","documentation":"StartupException with code ERR_WRONG_MACHINE_STATE thrown by DataResurrectionCheck.execute when the node's recorded heartbeat shows its last known uptime is older than gc_grace_seconds for some tables. Starting the node in that state could resurrect deleted data ('zombies'), so Cassandra refuses to start and asks for manual resolution and removal of the heartbeat file.","triggerScenarios":"A node was down longer than the smallest gc_grace_seconds among its tables and a heartbeat file from a previous run exists showing lastHeartbeat older than that grace window; executing the check during startup finds invalidTables non-empty and throws StartupException(ERR_WRONG_MACHINE_STATE, ...).","commonSituations":"Nodes that crashed or were stopped for longer than gc_grace (often low in CDC/tunable-consistency setups) then restarted; clocks/heartbeat file restored from an old snapshot; test clusters with very low gc_grace_seconds left idle over a weekend.","solutions":["Manually verify data consistency (run repair on the affected tables from a healthy node) before restarting.","Resolve the flagged tables (repair/anticompaction or re-bootstrap the node), then delete the stale heartbeat file and start the node.","If the tables are disposable (dev/test), truncate or drop them and remove the heartbeat, then restart.","Ensure nodes are not down longer than gc_grace_seconds; increase gc_grace_seconds or add monitoring/alerting on node downtime."],"exampleFix":"// before (force start, unsafe)\nrm heartbeat && ./cassandra -R\n// after\nnodetool repair -- myks tbl_with_low_gc_grace\n# after repair completes\ncassandra_stored_heartbeat=$(find /var/lib/cassandra -name '*heartbeat*')\nrm \"$cassandra_stored_heartbeat\" && ./cassandra","handlingStrategy":"validation","validationCode":"// Before restart, compare downtime vs gc_grace of all tables on the node:\nlong downtimeMs = System.currentTimeMillis() - lastKnownHeartbeatMs;\nif (downtimeMs > minGcGraceSecondsOfLocalTables * 1000L)\n    runRepairFirst(); // repair affected tables before starting the node","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Alert on node downtime approaching min(gc_grace_seconds) across local tables.","Keep gc_grace_seconds comfortably above expected max downtime.","Never delete the heartbeat file and force-start without repairing first.","After long outages, plan repair before restart as a runbook step."],"tags":["startup-check","data-consistency","zombie-data","gc-grace"],"backgroundTag":"invalid-state-transition","analyzedSha":"88fd0f6a0eaed8943f05ac9e8f947882b8ddc8f1","analyzedAt":"2026-09-10T07:29:22.284Z","contentChangedAt":"2026-09-10T07:29:22.284Z","schemaVersion":2},"datasetVersion":"2026-09-17T15:17:12.973Z"}