aeron-io/aeron · error · AgentTerminationException

failed to update recording log

Error message

failed to update recording log

What it means

ClusterBackupAgent throws AgentTerminationException("failed to update recording log") when an unexpected exception occurs while the backup agent checks/updates the recording log during a backup cycle. The original exception is reported via the counted error handler and the agent terminates, because a consistent recording log view is essential to continue backup safely.

Solutions

1. Inspect the exception logged by countedErrorHandler just before this message for the root cause
2. Check disk health and archive storage on the backup node
3. Restart the backup node / aeron archive to rebuild a consistent recording log state
4. Verify archive client connectivity and versions match the cluster

Example fix

// before
// backup agent crashes with AgentTerminationException on transient IO error
// after
// fix underlying storage, then restart:
// java -Daeron.cluster.backup.progress.timeout=... io.aeron.cluster.ClusterBackup
// and ensure the archive media is writable and healthy

// preflight: check archive reachable and disk healthy before starting backup
if (!archiveProxy.isConnected()) throw new IllegalStateException("archive not connected");

try {
    backup.start();
} catch (AgentTerminationException e) {
    log.error("backup agent terminated; check countedErrorHandler log for root cause", e);
    // alert ops, inspect disk/archive, then restart the backup node
}

When it happens

Trigger: Any exception thrown inside ClusterBackupAgent.doWork()'s recording-log update phase (e.g. AeronIOException reading recordings, rethrow of failures from the archive client while listing or extending recordings).

Common situations: Archive media failures/disk errors while querying the recording log, corrupted local recording catalog, or an archive client returning error responses mid-scan in a backup node process.

Understand the failure class

Background: "failed to read file", EACCES, ENOENT and "could not read <path>" errors: when a program can't read a file from disk — this error's family across 49 libraries.

Related errors

• failed to read link file=" + linkFile
• [clientId= , clientName= ] Failed to map log buffer with…
• UncheckedIOException while backing up recording log
• failed to read link file=
• failed to write recording

AI-assisted analysis of aeron-io/aeron@6d60124e15 (2026-09-12). Data as JSON: /api/errors/64a35eb1210a1dad. Report an issue: GitHub.

Appendix: source

Thrown at aeron-cluster/src/main/java/io/aeron/cluster/ClusterBackupAgent.java:971

                wasRecordingLogUpdated = true;
            }

            if (null != leaderLastTermEntry && recordingLog.isUnknown(leaderLastTermEntry.leadershipTermId))
            {
                recordingLog.appendTerm(
                    liveLogRecordingId,
                    leaderLastTermEntry.leadershipTermId,
                    leaderLastTermEntry.termBaseLogPosition,
                    leaderLastTermEntry.timestamp);

                wasRecordingLogUpdated = true;
                leaderLastTermEntry = null;
            }
        }
        catch (final Exception ex)
        {
            ctx.countedErrorHandler().onError(ex);
            throw new AgentTerminationException("failed to update recording log");
        }

        if (wasRecordingLogUpdated)
        {
            recordingLog.force(2);
            if (!snapshotsRetrieved.isEmpty())
            {
                ctx.snapshotRetrieveCounter().incrementRelease();
            }

            if (null != eventsListener)
            {
                eventsListener.onUpdatedRecordingLog(recordingLog, snapshotsRetrieved);
            }
        }

        snapshotsRetrieved.clear();
        snapshotsToRetrieve.clear();

View on GitHub (pinned to 6d60124e15)