{"record":{"id":"4209cb1c2f470452","repo":"benbjohnson/litestream","slug":"apply-gap-fill-ltx-file-level-d-min-s-max-s","errorCode":null,"errorMessage":"apply gap-fill ltx file (level=%d, min=%s, max=%s): %w","messagePattern":"apply gap-fill ltx file \\(level=(.+?), min=(.+?), max=(.+?)\\): %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"replica.go","lineNumber":1037,"sourceCode":"\t\t\t}\n\t\t\treturn currentTXID, retErr\n\t\t}\n\n\t\tfor itr.Next() {\n\t\t\tinfo := itr.Item()\n\n\t\t\t// Skip if there's a gap at this level too.\n\t\t\tif info.MinTXID > currentTXID+1 {\n\t\t\t\tbreak\n\t\t\t}\n\n\t\t\t// Skip if already covered.\n\t\t\tif info.MaxTXID <= currentTXID {\n\t\t\t\tcontinue\n\t\t\t}\n\n\t\t\tif err := r.applyLTXFile(ctx, f, info, pageSize); err != nil {\n\t\t\t\treturn closeLevel(fmt.Errorf(\n\t\t\t\t\t\"apply gap-fill ltx file (level=%d, min=%s, max=%s): %w\",\n\t\t\t\t\tinfo.Level, info.MinTXID, info.MaxTXID, err,\n\t\t\t\t))\n\t\t\t}\n\t\t\tcurrentTXID = info.MaxTXID\n\n\t\t\t// If we've bridged past the gap, we're done.\n\t\t\tif currentTXID+1 >= gapMinTXID {\n\t\t\t\treturn closeLevel(nil)\n\t\t\t}\n\t\t}\n\n\t\tif iterErr := itr.Err(); iterErr != nil {\n\t\t\treturn closeLevel(fmt.Errorf(\"iterate level %d ltx files: %w\", level, iterErr))\n\t\t}\n\t\tif _, err := closeLevel(nil); err != nil {\n\t\t\treturn currentTXID, err\n\t\t}","sourceCodeStart":1019,"sourceCodeEnd":1055,"githubUrl":"https://github.com/benbjohnson/litestream/blob/4ed7a308f6271ebfd2b0a6e4b70b03011a37e4a3/replica.go#L1019-L1055","documentation":"fillFollowGap found a higher-level LTX file that covers the TXID gap, but applying it via applyLTXFile failed. The error wraps the inner apply error (open/decode/write/checksum from errors 430-434) with the level and TXID range of the gap-fill file, so you can identify exactly which object failed.","triggerScenarios":"Any applyLTXFile failure (open ltx file, decode header, decode page, write page, sync, truncate, close decoder) while applying a level 1-3 file whose MinTXID <= currentTXID+1 and MaxTXID > currentTXID during follow-mode gap repair.","commonSituations":"Corrupt or truncated compacted LTX object in remote storage, disk full on the local DB volume mid-gap-fill, network interruption while streaming the LTX body, permission problems on the local database file.","solutions":["Read the wrapped cause to identify the stage that failed (open/decode/write/sync/truncate/close)","If the cause is checksum/decode: remove the corrupt object from storage and re-compact or re-snapshot","Run `litestream reset <db>` to clear local state and restore from a fresh snapshot","Check local disk space and write permissions before retrying"],"exampleFix":"# before: unknown failure on gap fill\nreplica: apply gap-fill ltx file (level=2, min=0000000000000010, max=0000000000000020): decode page: unexpected EOF\n# after: remove corrupt object and reset\naws s3 rm s3://bucket/db/0002/0000000000000010-0000000000000020.ltx\nlitestream reset /var/lib/db/app.db","handlingStrategy":"fallback","validationCode":"// sanity-check local env before gap-fill\nfunc precheckApply(dbPath string, pageSize uint32, info *ltx.FileInfo) error {\n    if fi, err := os.Stat(dbPath); err != nil || fi.Mode().Perm()&0200 == 0 {\n        return fmt.Errorf(\"db not writable: %w\", err)\n    }\n    if freeSpace(filepath.Dir(dbPath)) < int64(info.Size) {\n        return fmt.Errorf(\"insufficient space for %s\", info.Path())\n    }\n    return nil\n}","typeGuard":null,"tryCatchPattern":"if err := r.applyNewLTXFiles(ctx, f, pageSize); err != nil {\n    if strings.Contains(err.Error(), \"apply gap-fill ltx file\") {\n        // parse level/min/max from message; decide reset vs retry\n        if isChecksumLike(err) {\n            exec.Command(\"litestream\", \"reset\", dbPath).Run()\n        }\n        return fmt.Errorf(\"gap-fill failed: %w\", err)\n    }\n    return err\n}","preventionTips":["Alert on replication lag/gaps so gap-fill runs rarely and is well observed","Keep free disk headroom on the DB volume for gap-fill rewrites","Detect and remove corrupt compacted objects proactively via checksum audits","Document `litestream reset` as the recovery step for unbridgable gaps"],"tags":["ltx","gap-fill","replication","corruption"],"backgroundTag":"checksum-mismatch","analyzedSha":"4ed7a308f6271ebfd2b0a6e4b70b03011a37e4a3","analyzedAt":"2026-09-06T18:29:25.564Z","contentChangedAt":"2026-09-06T18:29:25.564Z","schemaVersion":2},"datasetVersion":"2026-09-14T00:17:10.932Z"}