microsoft/garnet · critical · Exception

ReplicaSyncSession replicaAofTail {replicaAofTailAddress} <

Error message

ReplicaSyncSession replicaAofTail {replicaAofTailAddress} < canServeFromAofAddress {checkpointAofBeginAddress}

What it means

Thrown during ReplicaSyncSession when, for a given sublog, the replica's AOF tail address is behind the checkpoint's begin address (replicaAofTailAddress[sublogIdx] < checkpointAofBeginAddress[sublogIdx]). This is an inconsistency: the checkpoint expects AOF data that the replica no longer has. The check is suppressed when serverOptions.FastAofTruncate is true (MainMemoryReplication mode where truncation is expected).

Source

Thrown at libs/server/AOF/GarnetAppendOnlyFile.cs:204

                ref AofAddress checkpointAofBeginAddress)
            {
                if (!recoverFromRemote)
                {
                    if (replicaAofBeginAddress[sublogIdx] > kFirstValidAofAddress && replicaAofBeginAddress[sublogIdx] > checkpointAofBeginAddress[sublogIdx])
                    {
                        logger?.LogInformation(
                            "ReplicaSyncSession: replicaAofBeginAddress {replicaAofBeginAddress} > PrimaryCheckpointRecoveredReplicationOffset {RecoveredReplicationOffset}, cannot use remote AOF",
                            replicaAofBeginAddress[sublogIdx], checkpointAofBeginAddress[sublogIdx]);
                    }
                    else
                    {
                        // Tail address cannot be behind the recovered address since above we checked replicaAofBeginAddress and it appears after RecoveredReplicationOffset
                        // unless we are performing MainMemoryReplication
                        // TODO: shouldn't we use the remote cEntry's tail address here since replica will recover to that?
                        if (replicaAofTailAddress[sublogIdx] < checkpointAofBeginAddress[sublogIdx] && !serverOptions.FastAofTruncate)
                        {
                            logger?.LogCritical("ReplicaSyncSession replicaAofTail {replicaAofTailAddress} < canServeFromAofAddress {RecoveredReplicationOffset}", replicaAofTailAddress, checkpointAofBeginAddress);
                            throw new Exception($"ReplicaSyncSession replicaAofTail {replicaAofTailAddress} < canServeFromAofAddress {checkpointAofBeginAddress}");
                        }

                        // If we are behind this primary we need to decide until where to replay
                        var replayUntilAddress = replicaAofTailAddress;
                        // Replica tail is further ahead than committed address of primary
                        if (Log.CommittedUntilAddress[sublogIdx] < replayUntilAddress[sublogIdx])
                            replayUntilAddress[sublogIdx] = Log.CommittedUntilAddress[sublogIdx];

                        // Replay only if records not included in checkpoint
                        if (replayUntilAddress[sublogIdx] > checkpointAofBeginAddress[sublogIdx])
                        {
                            logger?.LogInformation("ReplicaSyncSession: have to replay remote AOF from {beginAddress} until {untilAddress}", checkpointAofBeginAddress[sublogIdx], replayUntilAddress);
                            replayAOFMap |= 1UL << sublogIdx;
                            // Bound replayUntilAddress to ReplicationOffset2 to avoid replaying divergent history only if connecting replica was attached to old primary
                            if (sameHistory2 && replayUntilAddress[sublogIdx] > replicationOffset2[sublogIdx])
                                replayUntilAddress[sublogIdx] = replicationOffset2[sublogIdx];
                            checkpointAofBeginAddress = replayUntilAddress;
                        }

View on GitHub (pinned to 951b0fc683)

Solutions

1. Re-bootstrap the replica from a full checkpoint/snapshot (AOF addresses will then align).
2. If running MainMemoryReplication, ensure serverOptions.FastAofTruncate is set to true so this check is skipped by design.
3. Increase AOF retention (lower truncation aggressiveness) so the AOF tail stays behind the checkpoint begin address.
4. Verify checkpoint and AOF truncation policies are configured consistently across the cluster.

// Operator-level guard: before replica attach, verify AOF tail >= checkpoint begin
// If using MainMemoryReplication, ensure FastAofTruncate is enabled so this check is bypassed.
// In server options:
//   FastAofTruncate = true   (for MainMemoryReplication topologies)
// Otherwise, ensure AOF retention keeps the tail behind the checkpoint begin address.

When it happens

Trigger: During replica attach/sync, the AOF on the primary has been truncated (tail moved forward) beyond what the replica's checkpoint recovery expects. Happens when checkpoint history diverges or AOF truncation ran ahead of checkpoint retention, and FastAofTruncate is not enabled.

Common situations: A replica reconnecting after a long disconnect where the primary truncated its AOF; a checkpoint whose begin address references already-truncated AOF; mismatched checkpoint-retention and AOF-truncation settings; failover scenarios where the new primary's history diverges from the replica.

Related errors

• Failed syncing because replica requested truncated AOF addre
• Option {fileType} not supported
• Checkpoint history unavailable, need full checkpoint for {en
• Failed to validate main store metadata at insertion
• RetrieveCheckpointFile: unexpected state{retStateType}

AI-assisted analysis of microsoft/garnet@951b0fc683 (2026-08-13). Data as JSON: /api/errors/98c6a012a76b9a32. Report an issue: GitHub.