{"record":{"id":"9899ca0622d8c377","repo":"microsoft/garnet","slug":"failed-syncing-because-replica-requested-truncated","errorCode":null,"errorMessage":"Failed syncing because replica requested truncated AOF address","messagePattern":"Failed syncing because replica requested truncated AOF address","errorType":"exception","errorClass":"Exception","httpStatus":null,"severity":"error","filePath":"libs/server/AOF/GarnetAppendOnlyFile.cs","lineNumber":253,"sourceCode":"\n        /// <summary>\n        /// Perform a data loss check at recovery\n        /// </summary>\n        /// <param name=\"possibleAofDataLoss\"></param>\n        /// <param name=\"syncFromAofAddress\"></param>\n        /// <param name=\"logger\"></param>\n        /// <exception cref=\"Exception\"></exception>\n        public void DataLossCheck(bool possibleAofDataLoss, AofAddress syncFromAofAddress, ILogger logger = null)\n        {\n            var beginAddress = Log.BeginAddress;\n            var anyLesser = syncFromAofAddress.AnyLesser(beginAddress);\n\n            if (anyLesser)\n            {\n                if (!possibleAofDataLoss)\n                {\n                    logger?.LogError(\"syncFromAofAddress: {syncFromAofAddress} < beginAofAddress: {storeWrapper.appendOnlyFile.BeginAddress}\", syncFromAofAddress, beginAddress);\n                    throw new Exception(\"Failed syncing because replica requested truncated AOF address\");\n                }\n                else\n                {\n                    logger?.LogWarning(\"AOF truncated, unsafe attach: syncFromAofAddress: {syncFromAofAddress} < beginAofAddress: {storeWrapper.appendOnlyFile.BeginAddress}\", syncFromAofAddress, beginAddress);\n                }\n            }\n        }\n    }\n}","sourceCodeStart":235,"sourceCodeEnd":262,"githubUrl":"https://github.com/microsoft/garnet/blob/951b0fc6838721f89d102c2bbe1b914e8d39d700/libs/server/AOF/GarnetAppendOnlyFile.cs#L235-L262","documentation":"Thrown by GarnetAppendOnlyFile.DataLossCheck when a replica requests to sync from an AOF address that is below (lesser than) the primary's current AOF begin address — meaning the requested data has already been truncated. The throw only occurs when possibleAofDataLoss is false (the operator has not opted into unsafe attach); when true, a warning is logged instead and the attach proceeds with potential data loss.","triggerScenarios":"A replica requests syncFromAofAddress that points to truncated AOF data. DataLossCheck is called with possibleAofDataLoss=false (the default/safe mode). The replica's requested start address is behind Log.BeginAddress on any sublog.","commonSituations":"Replica was offline long enough for the primary to truncate the AOF past the replica's replication offset; a replica attaching to a new primary after failover where histories don't overlap; aggressive AOF truncation with short retention.","solutions":["Re-bootstrap the replica from a fresh full checkpoint so its replication offset is within the primary's current AOF range.","Increase AOF size/retention so truncation does not overtake replica lag.","If accepting data loss is intentional, pass possibleAofDataLoss=true (e.g. via the unsafe-attach server option) — but understand data will be missing.","Monitor replica lag and alert before it exceeds AOF retention."],"exampleFix":null,"handlingStrategy":"validation","validationCode":"// Before attaching a replica, check that its replication offset is within the primary's AOF range\n// (operator-level: compare replica's syncFromAofAddress against the primary's begin address).\n// If not, re-bootstrap the replica from a full checkpoint.\n// If data loss is acceptable, pass possibleAofDataLoss=true (unsafe-attach option).","typeGuard":null,"tryCatchPattern":"try { aof.DataLossCheck(possibleAofDataLoss: false, syncFromAofAddress, logger); }\ncatch (Exception ex) when (ex.Message.Contains(\"truncated AOF address\"))\n{\n    logger.LogError(\"Replica requested truncated AOF; re-bootstrap from full checkpoint or enable unsafe attach.\");\n    throw;\n}","preventionTips":["Increase AOF retention so truncation does not overtake replica lag.","Re-bootstrap stale replicas from a full checkpoint rather than incremental AOF sync.","Monitor replica lag and AOF begin-address movement.","Only enable unsafe attach (possibleAofDataLoss=true) with explicit acknowledgement of data loss."],"tags":["aof","replication","replica-sync","truncation"],"backgroundTag":null,"analyzedSha":"951b0fc6838721f89d102c2bbe1b914e8d39d700","analyzedAt":"2026-08-13T19:01:32.939Z","schemaVersion":2},"datasetVersion":"2026-08-13T19:17:28.613Z"}