benbjohnson/litestream · error

apply WAL segments: %w

Error message

apply WAL segments: %w

What it means

After the snapshot is downloaded, RestoreV3 replays the filtered WAL segments into the temp database via r.applyWALSegmentsV3; any failure is wrapped as 'apply WAL segments: %w'. This step rolls the snapshot forward to the latest (or requested timestamp) transaction; failures mean the restore cannot reach the target point-in-time.

Source

Thrown at replica.go:1152

	if db := r.DB(); db != nil {
		dirInfo = db.DirInfo()
	}
	if err := internal.MkdirAll(filepath.Dir(opt.OutputPath), dirInfo); err != nil {
		return fmt.Errorf("create parent directory: %w", err)
	}

	// Create temp file for restore.
	tmpPath := opt.OutputPath + ".tmp"
	defer func() { _ = os.Remove(tmpPath) }()

	// Download and decompress snapshot.
	if err := r.downloadSnapshotV3(ctx, client, snapshot.Generation, snapshot.Index, tmpPath); err != nil {
		return fmt.Errorf("download snapshot: %w", err)
	}

	// Apply WAL segments.
	if err := r.applyWALSegmentsV3(ctx, client, snapshot.Generation, snapshot.Index, segments, tmpPath); err != nil {
		return fmt.Errorf("apply WAL segments: %w", err)
	}

	// Rename to final path.
	if err := os.Rename(tmpPath, opt.OutputPath); err != nil {
		return fmt.Errorf("rename to output path: %w", err)
	}
	if err := internal.FsyncDir(filepath.Dir(opt.OutputPath)); err != nil {
		return fmt.Errorf("sync restore output dir: %w", err)
	}

	if opt.IntegrityCheck != IntegrityCheckNone {
		if err := checkIntegrity(ctx, opt.OutputPath, opt.IntegrityCheck); err != nil {
			if ctx.Err() == nil {
				_ = os.Remove(opt.OutputPath)
				_ = os.Remove(opt.OutputPath + "-shm")
				_ = os.Remove(opt.OutputPath + "-wal")
			}
			return fmt.Errorf("post-restore integrity check: %w", err)

View on GitHub (pinned to 4ed7a308f6)

Solutions

1. Inspect the wrapped cause to identify the failing segment (index/order/corruption)
2. Pick an earlier snapshot or later timestamp (RestoreOptions.Timestamp) that has a complete WAL chain
3. Verify WAL segment objects exist and are intact in the generation's WAL prefix; re-replicate if the replica is still live
4. Retry the restore to rule out transient network errors during segment downloads

Example fix

// before
var ts time.Time // latest — replay needs full WAL chain
err := replica.Restore(ctx, litestream.RestoreOptions{OutputPath: p, Timestamp: ts})
// after
snapshotTime := time.Date(2026, 9, 1, 12, 0, 0, 0, time.UTC) // target point with intact WALs
err := replica.Restore(ctx, litestream.RestoreOptions{OutputPath: p, Timestamp: snapshotTime})

// preflight: confirm a contiguous WAL chain exists for the snapshot's generation
segments, err := client.WALSegmentsV3(ctx, generation)
if err != nil {
    return err
}
filtered := litestream.FilterWALSegments(segments, snapshotIndex, timestamp) // or equivalent ordering check
if len(filtered) == 0 {
    return fmt.Errorf("no WAL segments available for replay")
}

if err := replica.Restore(ctx, opt); err != nil {
    if strings.Contains(err.Error(), "apply WAL segments") {
        log.Printf("WAL replay failed, cause=%v; trying earlier snapshot", errors.Unwrap(err))
        opt.Timestamp = opt.Timestamp.Add(-24 * time.Hour) // fall back to an earlier point
        return replica.Restore(ctx, opt)
    }
    return err
}

When it happens

Trigger: Calling Replica.Restore when applying WAL segments fails: a WAL segment object is missing or corrupted in storage, segments are out of order / gaps in the WAL chain, decompression of a segment fails, or a network error occurs while downloading segments mid-replay.

Common situations: Bucket lifecycle rules deleting old WAL segments needed by the replay; interrupted replication leaving gaps in the WAL chain; corrupted or truncated WAL objects; restore of a very old snapshot whose later WALs were compacted/removed under v0.3.x semantics.

Understand the failure class

Background: 'Could not be found', 'does not exist', 'not found in database': the resource-not-found family when an ID, slug, key, or URI lookup comes back empty — this error's family across 20 libraries.

Related errors

• list WAL segments: %w
• list generations: %w
• list snapshots for generation %s: %w
• download snapshot: %w
• no snapshots available

AI-assisted analysis of benbjohnson/litestream@4ed7a308f6 (2026-09-06). Data as JSON: /api/errors/f8a38dad74300bac. Report an issue: GitHub.