benbjohnson/litestream · error

remove expired l0 files: %w

Error message

remove expired l0 files: %w

What it means

Wraps a failure from DeleteLTXFiles when removing expired L0 files after retention enforcement computed the deletion list. The remote deletion failed (permissions, object lock, network); local cleanup still proceeds and remote deletion can be retried.

Source

Thrown at db.go:3124

	internal.L0RetentionGaugeVec.WithLabelValues(dbName, "eligible").Set(float64(len(deleted)))
	internal.L0RetentionGaugeVec.WithLabelValues(dbName, "not_compacted").Set(float64(notCompactedCount))
	internal.L0RetentionGaugeVec.WithLabelValues(dbName, "too_recent").Set(float64(tooRecentCount))

	db.Logger.Debug("l0 retention scan complete",
		"total_l0_files", totalFiles,
		"eligible_for_deletion", len(deleted),
		"not_compacted_yet", notCompactedCount,
		"too_recent", tooRecentCount,
		"max_l1_txid", maxL1TXID)

	if len(deleted) == 0 {
		return nil
	}

	if !db.RetentionEnabled {
		db.Logger.Debug("skipping remote deletion (retention disabled)", "level", 0, "count", len(deleted))
	} else if err := db.Replica.Client.DeleteLTXFiles(ctx, deleted); err != nil {
		return fmt.Errorf("remove expired l0 files: %w", err)
	}

	for _, info := range deleted {
		localPath := db.LTXPath(0, info.MinTXID, info.MaxTXID)
		db.Logger.Debug("deleting expired local l0 file", "minTXID", info.MinTXID, "maxTXID", info.MaxTXID, "path", localPath)
		if err := os.Remove(localPath); err != nil && !os.IsNotExist(err) {
			db.Logger.Error("failed to remove local l0 file", "path", localPath, "error", err)
		}
	}
	if len(deleted) > 0 {
		db.invalidatePosCache()
	}

	db.Logger.Info("l0 retention enforced", "deleted_count", len(deleted), "max_l1_txid", maxL1TXID)

	return nil
}

View on GitHub (pinned to 4ed7a308f6)

Solutions

1. Grant delete permission on the replica prefix to the credentials in use.
2. Check bucket policies (object lock, versioning, retention) blocking deletes.
3. Retry the retention cycle; deletions are idempotent.
4. Manually delete reported stuck keys if one object blocks the batch.
5. Disable litestream retention only if a storage lifecycle policy handles cleanup.

Example fix

// before
if err := db.Replica.Client.DeleteLTXFiles(ctx, deleted); err != nil { return fmt.Errorf("remove expired l0 files: %w", err) }
// after
if err := db.Replica.Client.DeleteLTXFiles(ctx, deleted); err != nil {
    if !isTransient(err) { return fmt.Errorf("remove expired l0 files: %w", err) } // else retry next cycle
}

// verify delete permission: aws iam simulate-principal-policy --action-names s3:DeleteObject

if err := db.Replica.Client.DeleteLTXFiles(ctx, deleted); err != nil {
    if isTransient(err) { scheduleRetry() } else { logAndAlert(fmt.Errorf("remove expired l0 files: %w", err)) }
}

When it happens

Trigger: L0 retention with RetentionEnabled=true deleting expired L0 files while the backend rejects or interrupts the delete: missing s3:DeleteObject permission, GCS retention policy, or a network failure mid-batch (db.go:3124).

Common situations: Least-privilege IAM policies allowing put/get but not delete; S3 versioned buckets with delete constraints; rate limits on bulk deletes.

Understand the failure class

Background: "failed to write file", "Could not save figure", "Error saving remote file" — file write failed: causes and fixes across languages and libraries — this error's family across 38 libraries.

Related errors

• remove ltx files: %w
• remove ltx files: %w
• abs: cannot delete ltx file %q: %w
• abs: cannot list blobs: %w
• abs: cannot delete blob %q: %w

AI-assisted analysis of benbjohnson/litestream@4ed7a308f6 (2026-09-06). Data as JSON: /api/errors/e733c9f10a81bcf2. Report an issue: GitHub.