benbjohnson/litestream · error

apply gap-fill ltx file (level=%d, min=%s, max=%s): %w

Error message

apply gap-fill ltx file (level=%d, min=%s, max=%s): %w

What it means

fillFollowGap found a higher-level LTX file that covers the TXID gap, but applying it via applyLTXFile failed. The error wraps the inner apply error (open/decode/write/checksum from errors 430-434) with the level and TXID range of the gap-fill file, so you can identify exactly which object failed.

Source

Thrown at replica.go:1037

			}
			return currentTXID, retErr
		}

		for itr.Next() {
			info := itr.Item()

			// Skip if there's a gap at this level too.
			if info.MinTXID > currentTXID+1 {
				break
			}

			// Skip if already covered.
			if info.MaxTXID <= currentTXID {
				continue
			}

			if err := r.applyLTXFile(ctx, f, info, pageSize); err != nil {
				return closeLevel(fmt.Errorf(
					"apply gap-fill ltx file (level=%d, min=%s, max=%s): %w",
					info.Level, info.MinTXID, info.MaxTXID, err,
				))
			}
			currentTXID = info.MaxTXID

			// If we've bridged past the gap, we're done.
			if currentTXID+1 >= gapMinTXID {
				return closeLevel(nil)
			}
		}

		if iterErr := itr.Err(); iterErr != nil {
			return closeLevel(fmt.Errorf("iterate level %d ltx files: %w", level, iterErr))
		}
		if _, err := closeLevel(nil); err != nil {
			return currentTXID, err
		}

View on GitHub (pinned to 4ed7a308f6)

Solutions

1. Read the wrapped cause to identify the stage that failed (open/decode/write/sync/truncate/close)
2. If the cause is checksum/decode: remove the corrupt object from storage and re-compact or re-snapshot
3. Run `litestream reset <db>` to clear local state and restore from a fresh snapshot
4. Check local disk space and write permissions before retrying

Example fix

# before: unknown failure on gap fill
replica: apply gap-fill ltx file (level=2, min=0000000000000010, max=0000000000000020): decode page: unexpected EOF
# after: remove corrupt object and reset
aws s3 rm s3://bucket/db/0002/0000000000000010-0000000000000020.ltx
litestream reset /var/lib/db/app.db

// sanity-check local env before gap-fill
func precheckApply(dbPath string, pageSize uint32, info *ltx.FileInfo) error {
    if fi, err := os.Stat(dbPath); err != nil || fi.Mode().Perm()&0200 == 0 {
        return fmt.Errorf("db not writable: %w", err)
    }
    if freeSpace(filepath.Dir(dbPath)) < int64(info.Size) {
        return fmt.Errorf("insufficient space for %s", info.Path())
    }
    return nil
}

if err := r.applyNewLTXFiles(ctx, f, pageSize); err != nil {
    if strings.Contains(err.Error(), "apply gap-fill ltx file") {
        // parse level/min/max from message; decide reset vs retry
        if isChecksumLike(err) {
            exec.Command("litestream", "reset", dbPath).Run()
        }
        return fmt.Errorf("gap-fill failed: %w", err)
    }
    return err
}

When it happens

Trigger: Any applyLTXFile failure (open ltx file, decode header, decode page, write page, sync, truncate, close decoder) while applying a level 1-3 file whose MinTXID <= currentTXID+1 and MaxTXID > currentTXID during follow-mode gap repair.

Common situations: Corrupt or truncated compacted LTX object in remote storage, disk full on the local DB volume mid-gap-fill, network interruption while streaming the LTX body, permission problems on the local database file.

Understand the failure class

Background: Checksum mismatch errors: "checksum verification failed", "digest mismatch", "expected vs actual checksum" — what they mean and how to fix them — this error's family across 41 libraries.

Related errors

• extract timestamp from LTX header: %w
• decode page: %w
• invalid replica, checksum mismatch
• ltx file corrupted
• verify

AI-assisted analysis of benbjohnson/litestream@4ed7a308f6 (2026-09-06). Data as JSON: /api/errors/4209cb1c2f470452. Report an issue: GitHub.