benbjohnson/litestream · error

missing WAL segment: expected

Error message

missing WAL segment: expected %d/%d, got %d/%d

What it means

During applyWALSegmentsV3, consecutive segments within a WAL file must be contiguous: the next segment's offset must equal the running offset accumulated from previously appended bytes. A mismatch means bytes are missing between segments. Note the message prints 'expected' twice from seg.Index but the running offset vs seg.Offset differ.

Solutions

1. List the generation's WAL segments and locate the missing offset; re-upload it from another replica if available.
2. Restore from the latest snapshot without timestamp filtering to start from an intact chain.
3. Choose a different generation or fall back to snapshot-only data.
4. Check storage bucket lifecycle/expiration rules so segments are not deleted out from under a generation.
5. Re-initialize replication (litestream reset) so a new generation with complete WALs is produced.

// verify contiguity of segments within each WAL index before restoring
for _, runs := range groupByIndex(segs) {
    var off int64
    for _, s := range runs {
        if s.Offset != off { /* gap found: choose another snapshot */ }
        off += s.Size
    }
}

if err := restore(ctx, opt); err != nil && strings.Contains(err.Error(), "missing WAL segment") {
    // fall back to snapshot-only restore or another generation
}

When it happens

Trigger: RestoreV3 encounters a WAL segment list with a hole inside one WAL file (e.g. index N offset X exists but offset X+len is missing while offset Y>X exists) — from partial uploads, deleted objects, or a broken listing in the v0.3.x replica client.

Common situations: Manual cleanup of storage mid-generation; interrupted uploads in the legacy v0.3.x writer; storage lifecycle rules deleting individual segment files; copying replicas with an incomplete object set.

Understand the failure class

Background: Record Not Found Errors: "not found", RecordNotFound, and "was not found" — what they mean and how to fix them — this error's family across 28 libraries.

Related errors

• apply WAL segments
• list WAL segments
• non-contiguous ltx files: have up to
• acquire read lock
• cannot access output path

AI-assisted analysis of benbjohnson/litestream@4ed7a308f6 (2026-09-06). Data as JSON: /api/errors/2fb2c19621021cb4. Report an issue: GitHub.

Appendix: source

Thrown at replica.go:1294

	// Reconstruct each wal file, appending non-zero offsets, and apply them
	// to the db.
	for _, seg := range segments {
		if seg.Offset == 0 {
			// Apply the last WAL file, if any
			if err = applyLastWalFile(); err != nil {
				return err
			}
			if seg.Index != expectedIndex {
				return fmt.Errorf("missing WAL index: expected %d/0, got %d/%d", expectedIndex, seg.Index, seg.Offset)
			}
			offset = 0
			// Open a new WAL file
			if f, err = os.OpenFile(walPath, os.O_CREATE|os.O_WRONLY|os.O_TRUNC, 0644); err != nil {
				return err
			}
			expectedIndex++
		} else if seg.Offset != offset {
			return fmt.Errorf("missing WAL segment: expected %d/%d, got %d/%d", seg.Index, offset, seg.Index, seg.Offset)
		}
		if n, err := r.appendWALSegmentV3(ctx, client, generation, seg, f); err != nil {
			return fmt.Errorf("write WAL segment %d/%d: %w", seg.Index, seg.Offset, err)
		} else {
			offset += n
			r.Logger().Debug("wrote WAL segment", "generation", generation, "index", seg.Index, "offset", seg.Offset, "bytes", n)
		}
	}

	return applyLastWalFile()
}

// appendWALSegmentV3 appends the specified segment to an open WAL file f.
func (r *Replica) appendWALSegmentV3(ctx context.Context, client ReplicaClientV3, generation string, seg WALSegmentInfoV3, f *os.File) (int64, error) {
	// Download WAL segment.
	rc, err := client.OpenWALSegmentV3(ctx, generation, seg.Index, seg.Offset)
	if err != nil {
		return 0, err

View on GitHub (pinned to 4ed7a308f6)