{"record":{"id":"e733c9f10a81bcf2","repo":"benbjohnson/litestream","slug":"remove-expired-l0-files-w-e733c9","errorCode":null,"errorMessage":"remove expired l0 files: %w","messagePattern":"remove expired l0 files: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"db.go","lineNumber":3124,"sourceCode":"\tinternal.L0RetentionGaugeVec.WithLabelValues(dbName, \"eligible\").Set(float64(len(deleted)))\n\tinternal.L0RetentionGaugeVec.WithLabelValues(dbName, \"not_compacted\").Set(float64(notCompactedCount))\n\tinternal.L0RetentionGaugeVec.WithLabelValues(dbName, \"too_recent\").Set(float64(tooRecentCount))\n\n\tdb.Logger.Debug(\"l0 retention scan complete\",\n\t\t\"total_l0_files\", totalFiles,\n\t\t\"eligible_for_deletion\", len(deleted),\n\t\t\"not_compacted_yet\", notCompactedCount,\n\t\t\"too_recent\", tooRecentCount,\n\t\t\"max_l1_txid\", maxL1TXID)\n\n\tif len(deleted) == 0 {\n\t\treturn nil\n\t}\n\n\tif !db.RetentionEnabled {\n\t\tdb.Logger.Debug(\"skipping remote deletion (retention disabled)\", \"level\", 0, \"count\", len(deleted))\n\t} else if err := db.Replica.Client.DeleteLTXFiles(ctx, deleted); err != nil {\n\t\treturn fmt.Errorf(\"remove expired l0 files: %w\", err)\n\t}\n\n\tfor _, info := range deleted {\n\t\tlocalPath := db.LTXPath(0, info.MinTXID, info.MaxTXID)\n\t\tdb.Logger.Debug(\"deleting expired local l0 file\", \"minTXID\", info.MinTXID, \"maxTXID\", info.MaxTXID, \"path\", localPath)\n\t\tif err := os.Remove(localPath); err != nil && !os.IsNotExist(err) {\n\t\t\tdb.Logger.Error(\"failed to remove local l0 file\", \"path\", localPath, \"error\", err)\n\t\t}\n\t}\n\tif len(deleted) > 0 {\n\t\tdb.invalidatePosCache()\n\t}\n\n\tdb.Logger.Info(\"l0 retention enforced\", \"deleted_count\", len(deleted), \"max_l1_txid\", maxL1TXID)\n\n\treturn nil\n}\n","sourceCodeStart":3106,"sourceCodeEnd":3142,"githubUrl":"https://github.com/benbjohnson/litestream/blob/4ed7a308f6271ebfd2b0a6e4b70b03011a37e4a3/db.go#L3106-L3142","documentation":"Wraps a failure from DeleteLTXFiles when removing expired L0 files after retention enforcement computed the deletion list. The remote deletion failed (permissions, object lock, network); local cleanup still proceeds and remote deletion can be retried.","triggerScenarios":"L0 retention with RetentionEnabled=true deleting expired L0 files while the backend rejects or interrupts the delete: missing s3:DeleteObject permission, GCS retention policy, or a network failure mid-batch (db.go:3124).","commonSituations":"Least-privilege IAM policies allowing put/get but not delete; S3 versioned buckets with delete constraints; rate limits on bulk deletes.","solutions":["Grant delete permission on the replica prefix to the credentials in use.","Check bucket policies (object lock, versioning, retention) blocking deletes.","Retry the retention cycle; deletions are idempotent.","Manually delete reported stuck keys if one object blocks the batch.","Disable litestream retention only if a storage lifecycle policy handles cleanup."],"exampleFix":"// before\nif err := db.Replica.Client.DeleteLTXFiles(ctx, deleted); err != nil { return fmt.Errorf(\"remove expired l0 files: %w\", err) }\n// after\nif err := db.Replica.Client.DeleteLTXFiles(ctx, deleted); err != nil {\n    if !isTransient(err) { return fmt.Errorf(\"remove expired l0 files: %w\", err) } // else retry next cycle\n}","handlingStrategy":"try-catch","validationCode":"// verify delete permission: aws iam simulate-principal-policy --action-names s3:DeleteObject","typeGuard":null,"tryCatchPattern":"if err := db.Replica.Client.DeleteLTXFiles(ctx, deleted); err != nil {\n    if isTransient(err) { scheduleRetry() } else { logAndAlert(fmt.Errorf(\"remove expired l0 files: %w\", err)) }\n}","preventionTips":["Least-privilege policies must still include delete on the replica prefix.","Avoid versioning/lock configs that block deletes.","Monitor delete failures; local cleanup proceeds but remote grows."],"tags":["storage","deletion","retention","permissions"],"backgroundTag":"file-write-failed","analyzedSha":"4ed7a308f6271ebfd2b0a6e4b70b03011a37e4a3","analyzedAt":"2026-09-06T18:29:25.564Z","contentChangedAt":"2026-09-06T18:29:25.564Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}