apache/cassandra · critical · StartupException

ERR_WRONG_MACHINE_STATE

ERR_WRONG_MACHINE_STATE

Error message

There are tables for which gc_grace_seconds is older than the lastly known time Cassandra node was up based on its heartbeat %s with timestamp %s. Cassandra node will not start as it would likely introduce data consistency issues (zombies etc). Please resolve these issues manually, then remove the heartbeat and start the node again. Invalid tables: %s

What it means

StartupException with code ERR_WRONG_MACHINE_STATE thrown by DataResurrectionCheck.execute when the node's recorded heartbeat shows its last known uptime is older than gc_grace_seconds for some tables. Starting the node in that state could resurrect deleted data ('zombies'), so Cassandra refuses to start and asks for manual resolution and removal of the heartbeat file.

Solutions

Manually verify data consistency (run repair on the affected tables from a healthy node) before restarting.
Resolve the flagged tables (repair/anticompaction or re-bootstrap the node), then delete the stale heartbeat file and start the node.
If the tables are disposable (dev/test), truncate or drop them and remove the heartbeat, then restart.
Ensure nodes are not down longer than gc_grace_seconds; increase gc_grace_seconds or add monitoring/alerting on node downtime.

Example fix

// before (force start, unsafe)
rm heartbeat && ./cassandra -R
// after
nodetool repair -- myks tbl_with_low_gc_grace
# after repair completes
cassandra_stored_heartbeat=$(find /var/lib/cassandra -name '*heartbeat*')
rm "$cassandra_stored_heartbeat" && ./cassandra

Defensive patterns

Strategy: validation

Validate before calling

// Before restart, compare downtime vs gc_grace of all tables on the node:
long downtimeMs = System.currentTimeMillis() - lastKnownHeartbeatMs;
if (downtimeMs > minGcGraceSecondsOfLocalTables * 1000L)
    runRepairFirst(); // repair affected tables before starting the node

Prevention

Alert on node downtime approaching min(gc_grace_seconds) across local tables.
Keep gc_grace_seconds comfortably above expected max downtime.
Never delete the heartbeat file and force-start without repairing first.
After long outages, plan repair before restart as a runbook step.

When it happens

Trigger: A node was down longer than the smallest gc_grace_seconds among its tables and a heartbeat file from a previous run exists showing lastHeartbeat older than that grace window; executing the check during startup finds invalidTables non-empty and throws StartupException(ERR_WRONG_MACHINE_STATE, ...).

Common situations: Nodes that crashed or were stopped for longer than gc_grace (often low in CDC/tunable-consistency setups) then restarted; clocks/heartbeat file restored from an old snapshot; test clusters with very low gc_grace_seconds left idle over a weekend.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

AI-assisted analysis of apache/cassandra@88fd0f6a0e (2026-09-10). Data as JSON: /api/errors/cbb87be496a8e7b7. Report an issue: GitHub.

Appendix: source

Thrown at src/java/org/apache/cassandra/service/DataResurrectionCheck.java:286

                    violations.add(Pair.create(keyspace, userTable.table));
            }
        }

        if (!violations.isEmpty())
        {
            String invalidTables = violations.stream()
                                             .map(p -> format("%s.%s", p.left, p.right))
                                             .collect(joining(","));

            String exceptionMessage = format("There are tables for which gc_grace_seconds is older " +
                                             "than the lastly known time Cassandra node was up based " +
                                             "on its heartbeat %s with timestamp %s. Cassandra node will not start " +
                                             "as it would likely introduce data consistency " +
                                             "issues (zombies etc). Please resolve these issues manually, " +
                                             "then remove the heartbeat and start the node again. Invalid tables: %s",
                                             heartbeatFile, heartbeat.lastHeartbeat, invalidTables);

            throw new StartupException(ERR_WRONG_MACHINE_STATE, exceptionMessage);
        }
    }

    @Override
    public void postAction(StartupChecksConfiguration configuration)
    {
        // Schedule heartbeating after all checks have passed, not as part of the check,
        // as it might happen that other checks after it might fail, but we would be heartbeating already.
        if (!configuration.isEnabled(name()))
            return;

        Map<String, Object> configMap = configuration.getConfig(name());
        File heartbeatFile = DataResurrectionCheck.getHeartbeatFile(configMap);

        ScheduledExecutors.scheduledTasks.scheduleAtFixedRate(() ->
        {
            Heartbeat heartbeat = new Heartbeat(Instant.ofEpochMilli(Clock.Global.currentTimeMillis()));
            try

View on GitHub (pinned to 88fd0f6a0e)