aeron-io/aeron · critical · ClusterEvent

no catchup progress: commitPosition=

Error message

no catchup progress: commitPosition=<commitPosition> limitPosition=<limitPosition> lastAppendPosition=<lastAppendPosition> appendPosition=<appendPosition> logPosition=<logPosition>

What it means

A follower in ACTIVE state making no catchup progress for longer than leaderHeartbeatTimeoutNs is considered stuck. The module throws ClusterEvent with a full position snapshot (commit/limit/append/log positions) to diagnose why catchup stalled.

Solutions

1. Use the message's position values to find where replay stalls (commit vs append vs limit)
2. Increase ctx.leaderHeartbeatTimeoutNs if the cluster legitimately pauses (e.g. snapshots, GC pauses)
3. Check follower disk I/O and network bandwidth against log production rate
4. Verify the leader is alive and publishing; restart the follower if the stall persists

Example fix

// before: aggressive timeout causing false stall detection
ctx.leaderHeartbeatTimeoutNs(TimeUnit.SECONDS.toNanos(5));

// after: tolerate slow replay
cCtx.leaderHeartbeatTimeoutNs(TimeUnit.SECONDS.toNanos(30));

// ensure replay throughput is plausible before starting
long timeoutNs = ctx.leaderHeartbeatTimeoutNs();
if (timeoutNs < TimeUnit.SECONDS.toNanos(10))
{
    throw new IllegalArgumentException("leaderHeartbeatTimeoutNs too low for catchup");
}

catch (ClusterEvent e)
{
    // message contains commit/limit/append/log positions
    log.error("catchup stalled, positions: {}", e.getMessage());
    metrics.recordCatchupStall(e.getMessage());
    throw e; // let AgentRunner restart the node
}

When it happens

Trigger: Thrown when nowNs exceeds timeOfLastAppendPositionUpdateNs + leaderHeartbeatTimeoutNs while state is ACTIVE and append position has not advanced.

Common situations: Slow disk preventing log replay; leader overloaded or paused; network throughput too low for replay rate; archive service bottleneck; heartbeat timeout configured too aggressively.

Understand the failure class

Background: Request timed out: what client-side request timeouts mean across libraries (Request timed out, TIMED_OUT, APITimeoutError) — this error's family across 39 libraries.

Related errors

• unexpected image close during catchup: position=
• failed to fetch remote recording descriptor
• failed to send recording position request
• failed to get recording position
• Archive connect timeout: step= publication= subscription=

AI-assisted analysis of aeron-io/aeron@6d60124e15 (2026-09-12). Data as JSON: /api/errors/287baeb378914f2e. Report an issue: GitHub.

Appendix: source

Thrown at aeron-cluster/src/main/java/io/aeron/cluster/ConsensusModuleAgent.java:2009

            if (0 == fragments && logAdapter.isImageClosed())
            {
                throw new ClusterEvent(
                    "unexpected image close during catchup: position=" + logAdapter.position());
            }

            workCount += updateFollowerPosition(
                election.leader().publication(),
                nowNs,
                leadershipTermId,
                currentAppendPosition,
                APPEND_POSITION_FLAG_CATCHUP);
            commitPosition.proposeMaxRelease(logAdapter.position());
        }

        if (nowNs > (timeOfLastAppendPositionUpdateNs + leaderHeartbeatTimeoutNs) &&
            ConsensusModule.State.ACTIVE == state)
        {
            throw new ClusterEvent(
                "no catchup progress:" +
                " commitPosition=" + commitPosition.getPlain() +
                " limitPosition=" + limitPosition +
                " lastAppendPosition=" + lastAppendPosition +
                " appendPosition=" + (null != appendPosition ? appendPosition.getPlain() : NULL_POSITION) +
                " logPosition=" + election.logPosition());
        }

        workCount += consensusModuleAdapter.poll();

        return workCount;
    }

    boolean isCatchupNearLive(final long position)
    {
        final Image image = logAdapter.image();
        if (null != image)
        {

View on GitHub (pinned to 6d60124e15)