{"record":{"id":"98c6a012a76b9a32","repo":"microsoft/garnet","slug":"replicasyncsession-replicaaoftail-replicaaoftaila","errorCode":null,"errorMessage":"ReplicaSyncSession replicaAofTail {replicaAofTailAddress} < canServeFromAofAddress {checkpointAofBeginAddress}","messagePattern":"ReplicaSyncSession replicaAofTail (.+?) < canServeFromAofAddress (.+?)","errorType":"exception","errorClass":"Exception","httpStatus":null,"severity":"critical","filePath":"libs/server/AOF/GarnetAppendOnlyFile.cs","lineNumber":204,"sourceCode":"                ref AofAddress checkpointAofBeginAddress)\n            {\n                if (!recoverFromRemote)\n                {\n                    if (replicaAofBeginAddress[sublogIdx] > kFirstValidAofAddress && replicaAofBeginAddress[sublogIdx] > checkpointAofBeginAddress[sublogIdx])\n                    {\n                        logger?.LogInformation(\n                            \"ReplicaSyncSession: replicaAofBeginAddress {replicaAofBeginAddress} > PrimaryCheckpointRecoveredReplicationOffset {RecoveredReplicationOffset}, cannot use remote AOF\",\n                            replicaAofBeginAddress[sublogIdx], checkpointAofBeginAddress[sublogIdx]);\n                    }\n                    else\n                    {\n                        // Tail address cannot be behind the recovered address since above we checked replicaAofBeginAddress and it appears after RecoveredReplicationOffset\n                        // unless we are performing MainMemoryReplication\n                        // TODO: shouldn't we use the remote cEntry's tail address here since replica will recover to that?\n                        if (replicaAofTailAddress[sublogIdx] < checkpointAofBeginAddress[sublogIdx] && !serverOptions.FastAofTruncate)\n                        {\n                            logger?.LogCritical(\"ReplicaSyncSession replicaAofTail {replicaAofTailAddress} < canServeFromAofAddress {RecoveredReplicationOffset}\", replicaAofTailAddress, checkpointAofBeginAddress);\n                            throw new Exception($\"ReplicaSyncSession replicaAofTail {replicaAofTailAddress} < canServeFromAofAddress {checkpointAofBeginAddress}\");\n                        }\n\n                        // If we are behind this primary we need to decide until where to replay\n                        var replayUntilAddress = replicaAofTailAddress;\n                        // Replica tail is further ahead than committed address of primary\n                        if (Log.CommittedUntilAddress[sublogIdx] < replayUntilAddress[sublogIdx])\n                            replayUntilAddress[sublogIdx] = Log.CommittedUntilAddress[sublogIdx];\n\n                        // Replay only if records not included in checkpoint\n                        if (replayUntilAddress[sublogIdx] > checkpointAofBeginAddress[sublogIdx])\n                        {\n                            logger?.LogInformation(\"ReplicaSyncSession: have to replay remote AOF from {beginAddress} until {untilAddress}\", checkpointAofBeginAddress[sublogIdx], replayUntilAddress);\n                            replayAOFMap |= 1UL << sublogIdx;\n                            // Bound replayUntilAddress to ReplicationOffset2 to avoid replaying divergent history only if connecting replica was attached to old primary\n                            if (sameHistory2 && replayUntilAddress[sublogIdx] > replicationOffset2[sublogIdx])\n                                replayUntilAddress[sublogIdx] = replicationOffset2[sublogIdx];\n                            checkpointAofBeginAddress = replayUntilAddress;\n                        }","sourceCodeStart":186,"sourceCodeEnd":222,"githubUrl":"https://github.com/microsoft/garnet/blob/951b0fc6838721f89d102c2bbe1b914e8d39d700/libs/server/AOF/GarnetAppendOnlyFile.cs#L186-L222","documentation":"Thrown during ReplicaSyncSession when, for a given sublog, the replica's AOF tail address is behind the checkpoint's begin address (replicaAofTailAddress[sublogIdx] < checkpointAofBeginAddress[sublogIdx]). This is an inconsistency: the checkpoint expects AOF data that the replica no longer has. The check is suppressed when serverOptions.FastAofTruncate is true (MainMemoryReplication mode where truncation is expected).","triggerScenarios":"During replica attach/sync, the AOF on the primary has been truncated (tail moved forward) beyond what the replica's checkpoint recovery expects. Happens when checkpoint history diverges or AOF truncation ran ahead of checkpoint retention, and FastAofTruncate is not enabled.","commonSituations":"A replica reconnecting after a long disconnect where the primary truncated its AOF; a checkpoint whose begin address references already-truncated AOF; mismatched checkpoint-retention and AOF-truncation settings; failover scenarios where the new primary's history diverges from the replica.","solutions":["Re-bootstrap the replica from a full checkpoint/snapshot (AOF addresses will then align).","If running MainMemoryReplication, ensure serverOptions.FastAofTruncate is set to true so this check is skipped by design.","Increase AOF retention (lower truncation aggressiveness) so the AOF tail stays behind the checkpoint begin address.","Verify checkpoint and AOF truncation policies are configured consistently across the cluster."],"exampleFix":null,"handlingStrategy":"validation","validationCode":"// Operator-level guard: before replica attach, verify AOF tail >= checkpoint begin\n// If using MainMemoryReplication, ensure FastAofTruncate is enabled so this check is bypassed.\n// In server options:\n//   FastAofTruncate = true   (for MainMemoryReplication topologies)\n// Otherwise, ensure AOF retention keeps the tail behind the checkpoint begin address.","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Enable FastAofTruncate when running MainMemoryReplication, where AOF truncation is expected.","Re-bootstrap replicas from a full checkpoint after long disconnects to realign AOF addresses.","Tune AOF retention and checkpoint frequency so the AOF tail stays behind checkpoint begin.","Monitor replica lag and alert before it risks truncation."],"tags":["aof","replication","replica-sync","data-loss"],"backgroundTag":null,"analyzedSha":"951b0fc6838721f89d102c2bbe1b914e8d39d700","analyzedAt":"2026-08-13T19:01:32.939Z","schemaVersion":2},"datasetVersion":"2026-08-13T19:17:28.613Z"}