benbjohnson/litestream · error

missing WAL segment: expected %d/%d, got %d/%d

Error message

missing WAL segment: expected %d/%d, got %d/%d

What it means

During applyWALSegmentsV3, consecutive segments within a WAL file must be contiguous: the next segment's offset must equal the running offset accumulated from previously appended bytes. A mismatch means bytes are missing between segments. Note the message prints 'expected' twice from seg.Index but the running offset vs seg.Offset differ.

Source

Thrown at replica.go:1294

	// Reconstruct each wal file, appending non-zero offsets, and apply them
	// to the db.
	for _, seg := range segments {
		if seg.Offset == 0 {
			// Apply the last WAL file, if any
			if err = applyLastWalFile(); err != nil {
				return err
			}
			if seg.Index != expectedIndex {
				return fmt.Errorf("missing WAL index: expected %d/0, got %d/%d", expectedIndex, seg.Index, seg.Offset)
			}
			offset = 0
			// Open a new WAL file
			if f, err = os.OpenFile(walPath, os.O_CREATE|os.O_WRONLY|os.O_TRUNC, 0644); err != nil {
				return err
			}
			expectedIndex++
		} else if seg.Offset != offset {
			return fmt.Errorf("missing WAL segment: expected %d/%d, got %d/%d", seg.Index, offset, seg.Index, seg.Offset)
		}
		if n, err := r.appendWALSegmentV3(ctx, client, generation, seg, f); err != nil {
			return fmt.Errorf("write WAL segment %d/%d: %w", seg.Index, seg.Offset, err)
		} else {
			offset += n
			r.Logger().Debug("wrote WAL segment", "generation", generation, "index", seg.Index, "offset", seg.Offset, "bytes", n)
		}
	}

	return applyLastWalFile()
}

// appendWALSegmentV3 appends the specified segment to an open WAL file f.
func (r *Replica) appendWALSegmentV3(ctx context.Context, client ReplicaClientV3, generation string, seg WALSegmentInfoV3, f *os.File) (int64, error) {
	// Download WAL segment.
	rc, err := client.OpenWALSegmentV3(ctx, generation, seg.Index, seg.Offset)
	if err != nil {
		return 0, err

View on GitHub (pinned to 4ed7a308f6)

Solutions

1. List the generation's WAL segments and locate the missing offset; re-upload it from another replica if available.
2. Restore from the latest snapshot without timestamp filtering to start from an intact chain.
3. Choose a different generation or fall back to snapshot-only data.
4. Check storage bucket lifecycle/expiration rules so segments are not deleted out from under a generation.
5. Re-initialize replication (litestream reset) so a new generation with complete WALs is produced.

// verify contiguity of segments within each WAL index before restoring
for _, runs := range groupByIndex(segs) {
    var off int64
    for _, s := range runs {
        if s.Offset != off { /* gap found: choose another snapshot */ }
        off += s.Size
    }
}

if err := restore(ctx, opt); err != nil && strings.Contains(err.Error(), "missing WAL segment") {
    // fall back to snapshot-only restore or another generation
}

When it happens

Trigger: RestoreV3 encounters a WAL segment list with a hole inside one WAL file (e.g. index N offset X exists but offset X+len is missing while offset Y>X exists) — from partial uploads, deleted objects, or a broken listing in the v0.3.x replica client.

Common situations: Manual cleanup of storage mid-generation; interrupted uploads in the legacy v0.3.x writer; storage lifecycle rules deleting individual segment files; copying replicas with an incomplete object set.

Understand the failure class

Background: Record Not Found Errors: "not found", RecordNotFound, and "was not found" — what they mean and how to fix them — this error's family across 28 libraries.

Related errors

• list WAL segments: %w
• apply WAL segments: %w
• non-contiguous ltx files: have up to %s but next file starts
• invalid -timestamp, must specify in ISO 8601 format (e.g. 20
• database already exists, skipping

AI-assisted analysis of benbjohnson/litestream@4ed7a308f6 (2026-09-06). Data as JSON: /api/errors/2fb2c19621021cb4. Report an issue: GitHub.