aeron-io/aeron · critical · ClusterException

recording has stopped unexpectedly

Error message

recording has stopped unexpectedly: <recordingId>

What it means

When waiting for a recording to reach a position, the agent watches the RecordingPos counter. If the counter becomes inactive before reaching the target position, the recording ended prematurely and ClusterException is thrown, since the cluster log can no longer advance safely.

Solutions

1. Check media driver and Archive logs for why the recording stopped (disk full, process exit)
2. Verify free disk space on the recording volume
3. Confirm the archive client session was not closed by application code
4. Restart the node; the cluster will recover from the last committed position or snapshot

Example fix

// before: no monitoring of recording health
// (recording dies silently until this exception)

// after: monitor RecordingPos counters and alert before position waits fail
if (!RecordingPos.isActive(counters, counterId, recordingId))
{
    logger.alert("recording stopped: " + recordingId);
}

// check recording health before awaiting a position
CountersReader counters = aeron.countersReader();
if (!RecordingPos.isActive(counters, counterId, recordingId))
{
    throw new IllegalStateException("recording no longer active: " + recordingId);
}

catch (ClusterException e)
{
    if (e.getMessage().startsWith("recording has stopped unexpectedly"))
    {
        log.error("log recording died; check driver/archive and disk space", e);
        // node must restart and recover from recording log
    }
    throw e;
}

When it happens

Trigger: Thrown from awaitRecordingPosition loop when RecordingPos.isActive(counters, counterId, recordingId) returns false while counters.getCounterValue(counterId) is still below the awaited position.

Common situations: Media driver or recording service died; archive session ended unexpectedly; disk full causing the recording to fail; node crash during log recording.

Understand the failure class

Background: Record Not Found Errors: "not found", RecordNotFound, and "was not found" — what they mean and how to fix them — this error's family across 28 libraries.

Related errors

• failed to fetch remote recording descriptor
• failed to send recording position request
• cannot live merge without active source recording
• failed to get recording position
• failed to resolve subscription endpoint: channel=" +…

AI-assisted analysis of aeron-io/aeron@6d60124e15 (2026-09-12). Data as JSON: /api/errors/46fc964cee47d1f7. Report an issue: GitHub.

Appendix: source

Thrown at aeron-cluster/src/main/java/io/aeron/cluster/ConsensusModuleAgent.java:3204

            recordingId, leadershipTermId, termBaseLogPosition, logPosition, timestamp, SERVICE_ID);

        recordingLog.force(ctx.fileSyncLevel());
        recoveryPlan = recordingLog.createRecoveryPlan(archive, serviceCount, Aeron.NULL_VALUE);
        totalSnapshotDurationTracker.onSnapshotEnd(clusterClock.timeNanos());
        ctx.snapshotCounter().incrementRelease();
    }

    private void awaitRecordingComplete(
        final long recordingId, final long position, final CountersReader counters, final int counterId)
    {
        idleStrategy.reset();
        while (counters.getCounterValue(counterId) < position)
        {
            idle();

            if (!RecordingPos.isActive(counters, counterId, recordingId))
            {
                throw new ClusterException("recording has stopped unexpectedly: " + recordingId);
            }
        }
    }

    private int awaitRecordingCounter(final CountersReader counters, final int sessionId, final long archiveId)
    {
        idleStrategy.reset();
        int counterId = RecordingPos.findCounterIdBySession(counters, sessionId, archiveId);
        while (CountersReader.NULL_COUNTER_ID == counterId)
        {
            idle();
            counterId = RecordingPos.findCounterIdBySession(counters, sessionId, archiveId);
        }

        return counterId;
    }

    private void snapshotState(

View on GitHub (pinned to 6d60124e15)