benbjohnson/litestream · error

write WAL segment %d/%d: %w

Error message

write WAL segment %d/%d: %w

What it means

Wraps any error from appendWALSegmentV3, which downloads one v0.3.x WAL segment from the replica client and appends it to the local WAL file being reconstructed during restore. The message identifies the failing segment as index/offset. The underlying cause is a download/IO failure, not a logical gap.

Source

Thrown at replica.go:1297

		if seg.Offset == 0 {
			// Apply the last WAL file, if any
			if err = applyLastWalFile(); err != nil {
				return err
			}
			if seg.Index != expectedIndex {
				return fmt.Errorf("missing WAL index: expected %d/0, got %d/%d", expectedIndex, seg.Index, seg.Offset)
			}
			offset = 0
			// Open a new WAL file
			if f, err = os.OpenFile(walPath, os.O_CREATE|os.O_WRONLY|os.O_TRUNC, 0644); err != nil {
				return err
			}
			expectedIndex++
		} else if seg.Offset != offset {
			return fmt.Errorf("missing WAL segment: expected %d/%d, got %d/%d", seg.Index, offset, seg.Index, seg.Offset)
		}
		if n, err := r.appendWALSegmentV3(ctx, client, generation, seg, f); err != nil {
			return fmt.Errorf("write WAL segment %d/%d: %w", seg.Index, seg.Offset, err)
		} else {
			offset += n
			r.Logger().Debug("wrote WAL segment", "generation", generation, "index", seg.Index, "offset", seg.Offset, "bytes", n)
		}
	}

	return applyLastWalFile()
}

// appendWALSegmentV3 appends the specified segment to an open WAL file f.
func (r *Replica) appendWALSegmentV3(ctx context.Context, client ReplicaClientV3, generation string, seg WALSegmentInfoV3, f *os.File) (int64, error) {
	// Download WAL segment.
	rc, err := client.OpenWALSegmentV3(ctx, generation, seg.Index, seg.Offset)
	if err != nil {
		return 0, err
	}
	defer func() { _ = rc.Close() }()

View on GitHub (pinned to 4ed7a308f6)

Solutions

1. Read the wrapped cause: network/403/404 from the backend vs local write error need different fixes.
2. Retry the restore — segment downloads are idempotent and transient network errors are common.
3. Verify storage credentials and object permissions for the bucket/prefix.
4. Free disk space where the output path lives; the reconstructed WAL can be large.
5. If a specific object is persistently missing/unreadable, restore from another generation.

Example fix

// before
os.Setenv("AWS_ACCESS_KEY_ID", "")
// after
export AWS_ACCESS_KEY_ID=... && litestream restore -output db.sqlite /path/db

var err error
for i := 0; i < 3; i++ {
    if err = replica.Restore(ctx, opt); err == nil ||
        !strings.Contains(err.Error(), "write WAL segment") {
        break
    }
    time.Sleep(backoff(i))
}

When it happens

Trigger: RestoreV3's replay loop calls r.appendWALSegmentV3(ctx, client, generation, seg, f) and it fails: network error fetching the segment from S3/GS/Azure/local storage, permission error opening the object, or write failure to the local tmp WAL file (disk full).

Common situations: Transient cloud-storage outages or throttling during restore; expired/missing storage credentials; local disk full while reconstructing a large WAL; proxy/firewall interrupting long restores.

Related errors

• download snapshot: %w
• abs: cannot delete ltx file %q: %w
• abs: cannot list blobs: %w
• abs: cannot delete blob %q: %w
• close iterator: %w

AI-assisted analysis of benbjohnson/litestream@4ed7a308f6 (2026-09-06). Data as JSON: /api/errors/73c356e6111ccff2. Report an issue: GitHub.