aeron-io/aeron · critical · AgentTerminationException

AgentTerminationException (storage space error on local…

Error message

AgentTerminationException (storage space error on local archive)

What it means

The backup agent polls the local archive for snapshot recording progress; if the archive returns an error with errorCode STORAGE_SPACE, the agent reports it and throws AgentTerminationException because backup cannot proceed without storage. Other archive errors are rethrown as-is.

Solutions

Free disk space on the backup node's archive storage
Increase the volume/quota backing the aeron archive data directory
Purge old/failed backup recordings from the archive
Add monitoring/alerting on disk usage before it fills

Example fix

// before
df -h # 100% on /var/lib/aeron/archive
// after
find /var/lib/aeron/archive -name '*.rec' -mtime +7 -delete
df -h # space reclaimed, restart backup

Defensive patterns

Strategy: validation

Validate before calling

File archiveDir = new File(ctx.archiveDirName());
if (archiveDir.getUsableSpace() < minRequiredBytes) {
    throw new IllegalStateException("insufficient archive storage: " + archiveDir.getUsableSpace());
}

Try / catch

try {
    backup.pollSnapshotProgress();
} catch (AgentTerminationException e) {
    // STORAGE_SPACE: free space or expand volume before restarting
}

Prevention

Alert at 70-80% disk usage on archive volumes
Schedule cleanup of obsolete backup recordings
Size volumes for the largest snapshot plus replay overhead

When it happens

Trigger: While retrieving a snapshot via the archive control poller, the poller surfaces an ArchiveException whose errorCode() == ArchiveException.STORAGE_SPACE.

Common situations: Backup node disk full (snapshot/replay writes exhausted space), undersized volume, leftover recordings filling the archive directory.

Understand the failure class

Background: "File too large" / "file size exceeds limit" errors: why libraries cap file sizes and how to fix them — this error's family across 46 libraries.

Related errors

AI-assisted analysis of aeron-io/aeron@6d60124e15 (2026-09-12). Data as JSON: /api/errors/ec65a537eabae2a2. Report an issue: GitHub.

Appendix: source

Thrown at aeron-cluster/src/main/java/io/aeron/cluster/ClusterBackupAgent.java:1073

        if (null != backupArchive)
        {
            final RecordingSignalPoller poller = this.recordingSignalPoller;
            workCount += poller.poll();

            if (poller.isPollComplete())
            {
                final int templateId = poller.templateId();

                if (ControlResponseDecoder.TEMPLATE_ID == templateId && poller.code() == ControlResponseCode.ERROR)
                {
                    final ArchiveException ex = new ArchiveException(
                        poller.errorMessage(), (int)poller.relevantId(), poller.correlationId());

                    if (ex.errorCode() == ArchiveException.STORAGE_SPACE)
                    {
                        ctx.countedErrorHandler().onError(ex);
                        throw new AgentTerminationException();
                    }
                    else
                    {
                        throw ex;
                    }
                }
                else if (RecordingSignalEventDecoder.TEMPLATE_ID == templateId && null != snapshotReplication)
                {
                    snapshotReplication.onSignal(
                        poller.correlationId(),
                        poller.recordingId(),
                        poller.recordingPosition(),
                        poller.recordingSignal());
                }
            }
            else if (0 == workCount && !poller.subscription().isConnected())
            {
                ctx.countedErrorHandler().onError(new ClusterEvent("local archive not connected"));

View on GitHub (pinned to 6d60124e15)