{"record":{"id":"287baeb378914f2e","repo":"aeron-io/aeron","slug":"no-catchup-progress-commitposition-commitposition","errorCode":null,"errorMessage":"no catchup progress: commitPosition=<commitPosition> limitPosition=<limitPosition> lastAppendPosition=<lastAppendPosition> appendPosition=<appendPosition> logPosition=<logPosition>","messagePattern":"no catchup progress: commitPosition=<commitPosition> limitPosition=<limitPosition> lastAppendPosition=<lastAppendPosition> appendPosition=<appendPosition> logPosition=<logPosition>","errorType":"exception","errorClass":"ClusterEvent","httpStatus":null,"severity":"critical","filePath":"aeron-cluster/src/main/java/io/aeron/cluster/ConsensusModuleAgent.java","lineNumber":2009,"sourceCode":"            if (0 == fragments && logAdapter.isImageClosed())\n            {\n                throw new ClusterEvent(\n                    \"unexpected image close during catchup: position=\" + logAdapter.position());\n            }\n\n            workCount += updateFollowerPosition(\n                election.leader().publication(),\n                nowNs,\n                leadershipTermId,\n                currentAppendPosition,\n                APPEND_POSITION_FLAG_CATCHUP);\n            commitPosition.proposeMaxRelease(logAdapter.position());\n        }\n\n        if (nowNs > (timeOfLastAppendPositionUpdateNs + leaderHeartbeatTimeoutNs) &&\n            ConsensusModule.State.ACTIVE == state)\n        {\n            throw new ClusterEvent(\n                \"no catchup progress:\" +\n                \" commitPosition=\" + commitPosition.getPlain() +\n                \" limitPosition=\" + limitPosition +\n                \" lastAppendPosition=\" + lastAppendPosition +\n                \" appendPosition=\" + (null != appendPosition ? appendPosition.getPlain() : NULL_POSITION) +\n                \" logPosition=\" + election.logPosition());\n        }\n\n        workCount += consensusModuleAdapter.poll();\n\n        return workCount;\n    }\n\n    boolean isCatchupNearLive(final long position)\n    {\n        final Image image = logAdapter.image();\n        if (null != image)\n        {","sourceCodeStart":1991,"sourceCodeEnd":2027,"githubUrl":"https://github.com/aeron-io/aeron/blob/6d60124e15e35c11b49ba2e3c2c2858a09a18803/aeron-cluster/src/main/java/io/aeron/cluster/ConsensusModuleAgent.java#L1991-L2027","documentation":"A follower in ACTIVE state making no catchup progress for longer than leaderHeartbeatTimeoutNs is considered stuck. The module throws ClusterEvent with a full position snapshot (commit/limit/append/log positions) to diagnose why catchup stalled.","triggerScenarios":"Thrown when nowNs exceeds timeOfLastAppendPositionUpdateNs + leaderHeartbeatTimeoutNs while state is ACTIVE and append position has not advanced.","commonSituations":"Slow disk preventing log replay; leader overloaded or paused; network throughput too low for replay rate; archive service bottleneck; heartbeat timeout configured too aggressively.","solutions":["Use the message's position values to find where replay stalls (commit vs append vs limit)","Increase ctx.leaderHeartbeatTimeoutNs if the cluster legitimately pauses (e.g. snapshots, GC pauses)","Check follower disk I/O and network bandwidth against log production rate","Verify the leader is alive and publishing; restart the follower if the stall persists"],"exampleFix":"// before: aggressive timeout causing false stall detection\nctx.leaderHeartbeatTimeoutNs(TimeUnit.SECONDS.toNanos(5));\n\n// after: tolerate slow replay\ncCtx.leaderHeartbeatTimeoutNs(TimeUnit.SECONDS.toNanos(30));","handlingStrategy":"validation","validationCode":"// ensure replay throughput is plausible before starting\nlong timeoutNs = ctx.leaderHeartbeatTimeoutNs();\nif (timeoutNs < TimeUnit.SECONDS.toNanos(10))\n{\n    throw new IllegalArgumentException(\"leaderHeartbeatTimeoutNs too low for catchup\");\n}","typeGuard":null,"tryCatchPattern":"catch (ClusterEvent e)\n{\n    // message contains commit/limit/append/log positions\n    log.error(\"catchup stalled, positions: {}\", e.getMessage());\n    metrics.recordCatchupStall(e.getMessage());\n    throw e; // let AgentRunner restart the node\n}","preventionTips":["Size leaderHeartbeatTimeoutNs above worst-case replay/snapshot time","Benchmark disk write throughput on followers","Alert on append position lag before the timeout fires","Keep leader and follower clock/GC pauses bounded"],"tags":["aeron","cluster","timeout","catchup","progress-stall"],"backgroundTag":"request-timeout","analyzedSha":"6d60124e15e35c11b49ba2e3c2c2858a09a18803","analyzedAt":"2026-09-12T11:17:07.683Z","contentChangedAt":"2026-09-12T11:17:07.683Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}