Failed to deserialize heartbeat file

Error message

Failed to deserialize heartbeat file {}. Falling back to file last modified time.

What it means

During execute(), the data-resurrection check reads a heartbeat JSON file; if it cannot be deserialized (IOException), the check falls back to using the file's last-modified timestamp as the heartbeat instant. The check still runs, but with a potentially less accurate heartbeat time.

Solutions

Inspect the heartbeat JSON file named in the log; delete it so a fresh one is written on next clean start, accepting the last-modified fallback
Fix JSON syntax errors if the file was hand-edited
Check file permissions/ownership and disk health for the heartbeat file's directory
Ensure clean shutdowns (nodetool drain/stop) so the heartbeat file is written atomically

Defensive patterns

Strategy: fallback

Validate before calling

// validate heartbeat file before the check reads it
File f = heartbeatFile;
if (!f.exists() || f.length() == 0 || !f.canRead())
    log.warn("Heartbeat file {} missing/empty/unreadable; expect last-modified fallback", f);

Try / catch

try {
    heartbeat = Heartbeat.deserializeFromJsonFile(heartbeatFile);
} catch (IOException ex) {
    LOGGER.warn("Failed to deserialize heartbeat file {}. Falling back to file last modified time.", heartbeatFile, ex);
    heartbeat = new Heartbeat(Instant.ofEpochMilli(heartbeatFile.lastModified()));
}

Prevention

Ensure clean shutdowns so the heartbeat JSON is fully written
Never hand-edit the heartbeat file; regenerate by clean restart
Verify write permissions and disk health of the heartbeat directory
Keep the heartbeat file on a local, non-network filesystem to avoid partial writes

When it happens

Trigger: Heartbeat.deserializeFromJsonFile(heartbeatFile) throws IOException: the heartbeat JSON is corrupt/truncated, empty, or unreadable due to permissions or concurrent writes during an unclean shutdown.

Common situations: Unclean node shutdown that left a partial heartbeat file; manual edits to the heartbeat file with invalid JSON; disk/permission issues on the heartbeat directory; version mismatch where the file format changed.

Understand the failure class

Background: "failed to unmarshal" / json.Unmarshal errors: why parsing a response into a Go struct fails and how to fix it — this error's family across 23 libraries.

Related errors

AI-assisted analysis of apache/cassandra@88fd0f6a0e (2026-09-10). Data as JSON: /api/errors/ab569894d8a94f13. Report an issue: GitHub.

Appendix: source

Thrown at src/java/org/apache/cassandra/service/DataResurrectionCheck.java:236

        Map<String, Object> config = configuration.getConfig(name());
        File heartbeatFile = getHeartbeatFile(config);

        if (!heartbeatFile.exists())
        {
            LOGGER.debug("Heartbeat file {} not found! Skipping heartbeat startup check.", heartbeatFile.absolutePath());
            return;
        }

        Heartbeat heartbeat;

        try
        {
            heartbeat = Heartbeat.deserializeFromJsonFile(heartbeatFile);
        }
        catch (IOException ex)
        {
            LOGGER.warn("Failed to deserialize heartbeat file {}. Falling back to file last modified time.",
                        heartbeatFile, ex);
            Instant lastModified = Instant.ofEpochMilli(heartbeatFile.lastModified());
            heartbeat = new Heartbeat(lastModified);
        }

        if (heartbeat.lastHeartbeat == null)
            return;

        long heartbeatMillis = heartbeat.lastHeartbeat.toEpochMilli();

        List<Pair<String, String>> violations = new ArrayList<>();

        Set<String> excludedKeyspaces = getExcludedKeyspaces(config);
        Set<Pair<String, String>> excludedTables = getExcludedTables(config);

        long minimumThresholdMillis = getMinimumThresholdMillis(config);

        long currentTimeMillis = currentTimeMillis();

View on GitHub (pinned to 88fd0f6a0e)