{"record":{"id":"46fc964cee47d1f7","repo":"aeron-io/aeron","slug":"recording-has-stopped-unexpectedly-recordingid","errorCode":null,"errorMessage":"recording has stopped unexpectedly: <recordingId>","messagePattern":"recording has stopped unexpectedly: <recordingId>","errorType":"exception","errorClass":"ClusterException","httpStatus":null,"severity":"critical","filePath":"aeron-cluster/src/main/java/io/aeron/cluster/ConsensusModuleAgent.java","lineNumber":3204,"sourceCode":"            recordingId, leadershipTermId, termBaseLogPosition, logPosition, timestamp, SERVICE_ID);\n\n        recordingLog.force(ctx.fileSyncLevel());\n        recoveryPlan = recordingLog.createRecoveryPlan(archive, serviceCount, Aeron.NULL_VALUE);\n        totalSnapshotDurationTracker.onSnapshotEnd(clusterClock.timeNanos());\n        ctx.snapshotCounter().incrementRelease();\n    }\n\n    private void awaitRecordingComplete(\n        final long recordingId, final long position, final CountersReader counters, final int counterId)\n    {\n        idleStrategy.reset();\n        while (counters.getCounterValue(counterId) < position)\n        {\n            idle();\n\n            if (!RecordingPos.isActive(counters, counterId, recordingId))\n            {\n                throw new ClusterException(\"recording has stopped unexpectedly: \" + recordingId);\n            }\n        }\n    }\n\n    private int awaitRecordingCounter(final CountersReader counters, final int sessionId, final long archiveId)\n    {\n        idleStrategy.reset();\n        int counterId = RecordingPos.findCounterIdBySession(counters, sessionId, archiveId);\n        while (CountersReader.NULL_COUNTER_ID == counterId)\n        {\n            idle();\n            counterId = RecordingPos.findCounterIdBySession(counters, sessionId, archiveId);\n        }\n\n        return counterId;\n    }\n\n    private void snapshotState(","sourceCodeStart":3186,"sourceCodeEnd":3222,"githubUrl":"https://github.com/aeron-io/aeron/blob/6d60124e15e35c11b49ba2e3c2c2858a09a18803/aeron-cluster/src/main/java/io/aeron/cluster/ConsensusModuleAgent.java#L3186-L3222","documentation":"When waiting for a recording to reach a position, the agent watches the RecordingPos counter. If the counter becomes inactive before reaching the target position, the recording ended prematurely and ClusterException is thrown, since the cluster log can no longer advance safely.","triggerScenarios":"Thrown from awaitRecordingPosition loop when RecordingPos.isActive(counters, counterId, recordingId) returns false while counters.getCounterValue(counterId) is still below the awaited position.","commonSituations":"Media driver or recording service died; archive session ended unexpectedly; disk full causing the recording to fail; node crash during log recording.","solutions":["Check media driver and Archive logs for why the recording stopped (disk full, process exit)","Verify free disk space on the recording volume","Confirm the archive client session was not closed by application code","Restart the node; the cluster will recover from the last committed position or snapshot"],"exampleFix":"// before: no monitoring of recording health\n// (recording dies silently until this exception)\n\n// after: monitor RecordingPos counters and alert before position waits fail\nif (!RecordingPos.isActive(counters, counterId, recordingId))\n{\n    logger.alert(\"recording stopped: \" + recordingId);\n}","handlingStrategy":"validation","validationCode":"// check recording health before awaiting a position\nCountersReader counters = aeron.countersReader();\nif (!RecordingPos.isActive(counters, counterId, recordingId))\n{\n    throw new IllegalStateException(\"recording no longer active: \" + recordingId);\n}","typeGuard":null,"tryCatchPattern":"catch (ClusterException e)\n{\n    if (e.getMessage().startsWith(\"recording has stopped unexpectedly\"))\n    {\n        log.error(\"log recording died; check driver/archive and disk space\", e);\n        // node must restart and recover from recording log\n    }\n    throw e;\n}","preventionTips":["Monitor RecordingPos counters as a health metric","Alert before disk fills; recordings fail when writes fail","Supervise the media driver and archive processes with auto-restart","Do not close archive sessions the cluster depends on"],"tags":["aeron","archive","recording","counter","replication"],"backgroundTag":"record-not-found","analyzedSha":"6d60124e15e35c11b49ba2e3c2c2858a09a18803","analyzedAt":"2026-09-12T11:17:07.683Z","contentChangedAt":"2026-09-12T11:17:07.683Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}