{"record":{"id":"bf7586ff3b0a2545","repo":"thanos-io/thanos","slug":"delete-from-source","errorCode":null,"errorMessage":"delete from source","messagePattern":"delete from source","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"pkg/verifier/safe_delete.go","lineNumber":79,"sourceCode":"\t\t}\n\t}()\n\n\t// Download the TSDB block.\n\tdir := filepath.Join(tempdir, id.String())\n\tif err := block.Download(ctx, ctx.Logger, ctx.Bkt, id, dir); err != nil {\n\t\treturn errors.Wrap(err, \"download from source\")\n\t}\n\n\t// Backup the block.\n\tif err := backupDownloaded(ctx, ctx.Logger, dir, ctx.BackupBkt, id); err != nil {\n\t\treturn err\n\t}\n\n\t// Block uploaded, so we are ok to remove from src bucket.\n\tif ctx.DeleteDelay.Seconds() == 0 {\n\t\tlevel.Info(ctx.Logger).Log(\"msg\", \"Deleting block\", \"id\", id.String())\n\t\tif err := block.Delete(ctx, ctx.Logger, ctx.Bkt, id); err != nil {\n\t\t\treturn errors.Wrap(err, \"delete from source\")\n\t\t}\n\t}\n\n\tlevel.Info(ctx.Logger).Log(\"msg\", \"Marking block as deleted\", \"id\", id.String())\n\tif err := block.MarkForDeletion(ctx, ctx.Logger, ctx.Bkt, id, \"manual verify-repair\", ctx.metrics.blocksMarkedForDeletion); err != nil {\n\t\treturn errors.Wrap(err, \"marking delete from source\")\n\t}\n\treturn nil\n}\n\n// BackupAndDeleteDownloaded works much like BackupAndDelete in that it will\n// move a TSDB block from a bucket to a backup bucket. If deleteDelay param is zero, block is removed from source bucket.\n// else the block is marked for deletion. The bdir parameter\n// points to the location on disk where the TSDB block was previously\n// downloaded allowing this function to avoid downloading the TSDB block from\n// the source bucket again. An error is returned if any operation fails.\nfunc BackupAndDeleteDownloaded(ctx Context, bdir string, id ulid.ULID) error {\n\t// Does this TSDB block exist in backupBkt already?","sourceCodeStart":61,"sourceCodeEnd":97,"githubUrl":"https://github.com/thanos-io/thanos/blob/35b8b991177def87ed52dcf10f9b6d87f07282c8/pkg/verifier/safe_delete.go#L61-L97","documentation":"BackupAndDelete moves a TSDB block to a backup bucket and then removes it from the source bucket. This error wraps a failure from block.Delete when deleting the block from the source bucket (only attempted when ctx.DeleteDelay is zero). It indicates the block was backed up but could not be removed from the origin bucket.","triggerScenarios":"Calling VerifyRepair which invokes BackupAndDelete with ctx.DeleteDelay == 0, after the block was successfully uploaded to the backup bucket, and block.Delete(ctx, ctx.Logger, ctx.Bkt, id) fails (e.g. bucket permission errors, object lock/retention, transient storage errors).","commonSituations":"Running thanos verify with --delete-delay=0 against a bucket where the credentials lack delete permission; S3/GCS object versioning or retention policies blocking deletes; transient object store outages during verify-repair.","solutions":["Check bucket credentials/IAM allow delete operations on the source bucket","Check for object-lock/retention or versioning policies preventing deletion","Re-run the verify-repair; the block is already backed up so the delete can be retried","Inspect the wrapped cause in the error chain for the underlying object-store error"],"exampleFix":"// before\nif ctx.DeleteDelay.Seconds() == 0 { ... block.Delete ... }\n// after\nif err := block.Delete(ctx, ctx.Logger, ctx.Bkt, id); err != nil {\n\tlevel.Warn(ctx.Logger).Log(\"msg\", \"failed to delete block from source; will be marked for deletion\", \"id\", id.String(), \"err\", err)\n\treturn errors.Wrap(err, \"delete from source\")\n}","handlingStrategy":"try-catch","validationCode":"// Go: pre-check delete permission if possible\nallowed, err := sourceBkt.Iter(ctx, \"\", func(string) error { return nil })\nif err != nil { /* bucket unreadable; expect delete problems too */ }\n_ = allowed","typeGuard":null,"tryCatchPattern":"if err := runVerifyRepair(); err != nil {\n\tif strings.Contains(err.Error(), \"delete from source\") {\n\t\t// backup succeeded; retry later or delete manually\n\t}\n}","preventionTips":["Use bucket credentials with delete permission","Set a non-zero delete-delay to use marking instead of hard delete","Watch object-lock/retention policies","Retry verify-repair after transient storage errors"],"tags":["object-storage","block-deletion","thanos"],"backgroundTag":"api-error-response","analyzedSha":"35b8b991177def87ed52dcf10f9b6d87f07282c8","analyzedAt":"2026-09-07T01:49:59.689Z","contentChangedAt":"2026-09-07T01:49:59.689Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}