{"record":{"id":"2fb2c19621021cb4","repo":"benbjohnson/litestream","slug":"missing-wal-segment-expected-d-d-got-d-d","errorCode":null,"errorMessage":"missing WAL segment: expected %d/%d, got %d/%d","messagePattern":"missing WAL segment: expected (.+?)/(.+?), got (.+?)/(.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"replica.go","lineNumber":1294,"sourceCode":"\t// Reconstruct each wal file, appending non-zero offsets, and apply them\n\t// to the db.\n\tfor _, seg := range segments {\n\t\tif seg.Offset == 0 {\n\t\t\t// Apply the last WAL file, if any\n\t\t\tif err = applyLastWalFile(); err != nil {\n\t\t\t\treturn err\n\t\t\t}\n\t\t\tif seg.Index != expectedIndex {\n\t\t\t\treturn fmt.Errorf(\"missing WAL index: expected %d/0, got %d/%d\", expectedIndex, seg.Index, seg.Offset)\n\t\t\t}\n\t\t\toffset = 0\n\t\t\t// Open a new WAL file\n\t\t\tif f, err = os.OpenFile(walPath, os.O_CREATE|os.O_WRONLY|os.O_TRUNC, 0644); err != nil {\n\t\t\t\treturn err\n\t\t\t}\n\t\t\texpectedIndex++\n\t\t} else if seg.Offset != offset {\n\t\t\treturn fmt.Errorf(\"missing WAL segment: expected %d/%d, got %d/%d\", seg.Index, offset, seg.Index, seg.Offset)\n\t\t}\n\t\tif n, err := r.appendWALSegmentV3(ctx, client, generation, seg, f); err != nil {\n\t\t\treturn fmt.Errorf(\"write WAL segment %d/%d: %w\", seg.Index, seg.Offset, err)\n\t\t} else {\n\t\t\toffset += n\n\t\t\tr.Logger().Debug(\"wrote WAL segment\", \"generation\", generation, \"index\", seg.Index, \"offset\", seg.Offset, \"bytes\", n)\n\t\t}\n\t}\n\n\treturn applyLastWalFile()\n}\n\n// appendWALSegmentV3 appends the specified segment to an open WAL file f.\nfunc (r *Replica) appendWALSegmentV3(ctx context.Context, client ReplicaClientV3, generation string, seg WALSegmentInfoV3, f *os.File) (int64, error) {\n\t// Download WAL segment.\n\trc, err := client.OpenWALSegmentV3(ctx, generation, seg.Index, seg.Offset)\n\tif err != nil {\n\t\treturn 0, err","sourceCodeStart":1276,"sourceCodeEnd":1312,"githubUrl":"https://github.com/benbjohnson/litestream/blob/4ed7a308f6271ebfd2b0a6e4b70b03011a37e4a3/replica.go#L1276-L1312","documentation":"During applyWALSegmentsV3, consecutive segments within a WAL file must be contiguous: the next segment's offset must equal the running offset accumulated from previously appended bytes. A mismatch means bytes are missing between segments. Note the message prints 'expected' twice from seg.Index but the running offset vs seg.Offset differ.","triggerScenarios":"RestoreV3 encounters a WAL segment list with a hole inside one WAL file (e.g. index N offset X exists but offset X+len is missing while offset Y>X exists) — from partial uploads, deleted objects, or a broken listing in the v0.3.x replica client.","commonSituations":"Manual cleanup of storage mid-generation; interrupted uploads in the legacy v0.3.x writer; storage lifecycle rules deleting individual segment files; copying replicas with an incomplete object set.","solutions":["List the generation's WAL segments and locate the missing offset; re-upload it from another replica if available.","Restore from the latest snapshot without timestamp filtering to start from an intact chain.","Choose a different generation or fall back to snapshot-only data.","Check storage bucket lifecycle/expiration rules so segments are not deleted out from under a generation.","Re-initialize replication (litestream reset) so a new generation with complete WALs is produced."],"exampleFix":null,"handlingStrategy":"validation","validationCode":"// verify contiguity of segments within each WAL index before restoring\nfor _, runs := range groupByIndex(segs) {\n    var off int64\n    for _, s := range runs {\n        if s.Offset != off { /* gap found: choose another snapshot */ }\n        off += s.Size\n    }\n}","typeGuard":null,"tryCatchPattern":"if err := restore(ctx, opt); err != nil && strings.Contains(err.Error(), \"missing WAL segment\") {\n    // fall back to snapshot-only restore or another generation\n}","preventionTips":["Prefer restoring the latest snapshot without timestamp filtering","Disable aggressive object-expiration rules on replica buckets","Copy replicas atomically (all objects of a generation together)","Run scheduled test restores to catch holes early"],"tags":["wal","restore","gap","contiguity"],"backgroundTag":"record-not-found","analyzedSha":"4ed7a308f6271ebfd2b0a6e4b70b03011a37e4a3","analyzedAt":"2026-09-06T18:29:25.564Z","contentChangedAt":"2026-09-06T18:29:25.564Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}