benbjohnson/litestream · error

open remote L0 file: %w

Error message

open remote L0 file: %w

What it means

After recreating the L0 directory, Litestream opens the remote L0 LTX file on the replica via db.Replica.Client.OpenLTXFile(ctx, 0, minTXID, maxTXID, 0, 0). This error wraps any failure to open that remote object, meaning the snapshot source file cannot be read from storage.

Source

Thrown at db.go:1627

	db.Logger.Info("detected database behind replica",
		"db_txid", dbPos.TXID,
		"replica_txid", replicaInfo.MaxTXID)

	// Clear local L0 files
	l0Dir := db.LTXLevelDir(0)
	if err := os.RemoveAll(l0Dir); err != nil && !os.IsNotExist(err) {
		return fmt.Errorf("remove L0 directory: %w", err)
	}
	db.invalidatePosCache()
	if err := internal.MkdirAll(l0Dir, db.dirInfo); err != nil {
		return fmt.Errorf("recreate L0 directory: %w", err)
	}

	// Fetch latest L0 LTX file from replica
	minTXID, maxTXID := replicaInfo.MinTXID, replicaInfo.MaxTXID
	reader, err := db.Replica.Client.OpenLTXFile(ctx, 0, minTXID, maxTXID, 0, 0)
	if err != nil {
		return fmt.Errorf("open remote L0 file: %w", err)
	}
	defer func() { _ = reader.Close() }()

	// Write to temp file and atomically rename
	localPath := db.LTXPath(0, minTXID, maxTXID)
	tmpPath := localPath + ".tmp"

	tmpFile, err := os.Create(tmpPath)
	if err != nil {
		return fmt.Errorf("create temp L0 file: %w", err)
	}
	defer func() { _ = os.Remove(tmpPath) }() // Clean up temp file on error

	if _, err := io.Copy(tmpFile, reader); err != nil {
		_ = tmpFile.Close()
		return fmt.Errorf("copy L0 file: %w", err)
	}

View on GitHub (pinned to 4ed7a308f6)

Solutions

1. Read the wrapped cause: a 404/NoSuchKey means the file disappeared (check bucket lifecycle/retention rules); 403 means missing read permission.
2. Restore read permissions (s3:GetObject / Storage Object Viewer) for the replica credentials.
3. If a lifecycle policy deletes L0 files, disable it or align retention so files referenced by Min/MaxTXID still exist.
4. Retry — many object-store failures are transient; the next sync re-runs the behind-check.

Example fix

// before: S3 policy allows List only
{"Effect":"Allow","Action":["s3:ListBucket"],"Resource":["arn:aws:s3:::mybucket"]}
// after: also allow object reads
{"Effect":"Allow","Action":["s3:GetObject"],"Resource":["arn:aws:s3:::mybucket/*"]}

// Verify the remote object is readable before relying on it
out, err := s3Client.GetObject(&s3.GetObjectInput{Bucket: b, Key: k})
if err != nil { log.Printf("remote LTX unreadable: %v", err) }

null

reader, err := client.OpenLTXFile(ctx, 0, minTXID, maxTXID, 0, 0)
if err != nil {
    var nf *ltx.NoSuchFileError
    if errors.As(err, &nf) { log.Printf("LTX file missing from replica: %v", err) }
    return err // litestream retries on next sync
}

When it happens

Trigger: checkDatabaseBehindReplica calls OpenLTXFile for level 0 with the replica's MinTXID..MaxTXID range and the storage client returns an error — object not found (the exact LTX file vanished from the bucket), 403 on GET, network failure mid-request, or unsupported/nil client.

Common situations: Retention/lifecycle rules deleted the remote L0 file between the MaxLTXFileInfo listing and the GET; wrong bucket/path after a config change; credentials allow listing but not reading (S3 s3:GetObject missing); transient object-store outage.

Understand the failure class

Background: 'Could not be found', 'does not exist', 'not found in database': the resource-not-found family when an ID, slug, key, or URI lookup comes back empty — this error's family across 20 libraries.

Related errors

• get replica position: %w
• copy L0 file: %w
• invalid replica, checksum mismatch
• must specify replica for database
• cannot specify 'replica' and 'replicas' on a database

AI-assisted analysis of benbjohnson/litestream@4ed7a308f6 (2026-09-06). Data as JSON: /api/errors/34a3d897e8d08167. Report an issue: GitHub.