{"record":{"id":"d0b3096d1e75b10d","repo":"thanos-io/thanos","slug":"marking-delete-from-source","errorCode":null,"errorMessage":"marking delete from source","messagePattern":"marking delete from source","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"pkg/verifier/safe_delete.go","lineNumber":85,"sourceCode":"\t\treturn errors.Wrap(err, \"download from source\")\n\t}\n\n\t// Backup the block.\n\tif err := backupDownloaded(ctx, ctx.Logger, dir, ctx.BackupBkt, id); err != nil {\n\t\treturn err\n\t}\n\n\t// Block uploaded, so we are ok to remove from src bucket.\n\tif ctx.DeleteDelay.Seconds() == 0 {\n\t\tlevel.Info(ctx.Logger).Log(\"msg\", \"Deleting block\", \"id\", id.String())\n\t\tif err := block.Delete(ctx, ctx.Logger, ctx.Bkt, id); err != nil {\n\t\t\treturn errors.Wrap(err, \"delete from source\")\n\t\t}\n\t}\n\n\tlevel.Info(ctx.Logger).Log(\"msg\", \"Marking block as deleted\", \"id\", id.String())\n\tif err := block.MarkForDeletion(ctx, ctx.Logger, ctx.Bkt, id, \"manual verify-repair\", ctx.metrics.blocksMarkedForDeletion); err != nil {\n\t\treturn errors.Wrap(err, \"marking delete from source\")\n\t}\n\treturn nil\n}\n\n// BackupAndDeleteDownloaded works much like BackupAndDelete in that it will\n// move a TSDB block from a bucket to a backup bucket. If deleteDelay param is zero, block is removed from source bucket.\n// else the block is marked for deletion. The bdir parameter\n// points to the location on disk where the TSDB block was previously\n// downloaded allowing this function to avoid downloading the TSDB block from\n// the source bucket again. An error is returned if any operation fails.\nfunc BackupAndDeleteDownloaded(ctx Context, bdir string, id ulid.ULID) error {\n\t// Does this TSDB block exist in backupBkt already?\n\tfound, err := TSDBBlockExistsInBucket(ctx, ctx.BackupBkt, id)\n\tif err != nil {\n\t\treturn err\n\t}\n\tif found {\n\t\treturn errors.Errorf(\"%s dir seems to exists in backup bucket. Remove this block manually if you are sure it is safe to do\", id)","sourceCodeStart":67,"sourceCodeEnd":103,"githubUrl":"https://github.com/thanos-io/thanos/blob/35b8b991177def87ed52dcf10f9b6d87f07282c8/pkg/verifier/safe_delete.go#L67-L103","documentation":"After backing up a block, BackupAndDelete marks it for deletion in the source bucket via block.MarkForDeletion when DeleteDelay is non-zero. This error wraps a failure of that mark operation, meaning the deletion marker (deletion-mark.json) could not be written to the source bucket.","triggerScenarios":"BackupAndDelete invoked with ctx.DeleteDelay > 0; the block is backed up but block.MarkForDeletion(ctx, ctx.Logger, ctx.Bkt, id, \"manual verify-repair\", ...) returns an error (write failure to source bucket).","commonSituations":"Read-only or partially-scoped bucket credentials missing write permission for deletion-mark.json; compactor concurrently holding/overwriting the mark; transient bucket errors during verify --repair runs.","solutions":["Grant write permission to the source bucket for deletion-mark.json objects","Ensure no conflicting compactor process is overriding the deletion mark","Retry verify-repair; block is safely backed up already","Check the wrapped error for the underlying object-store cause"],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"// ensure write access by touching a test object\nerr := markBkt.Upload(ctx, \"__perm_test__\", bytes.NewReader(nil))","typeGuard":null,"tryCatchPattern":"if err := runVerifyRepair(); err != nil {\n\tif strings.Contains(err.Error(), \"marking delete from source\") {\n\t\t// block backed up; fix bucket write perms and retry\n\t}\n}","preventionTips":["Grant write access for deletion-mark.json","Avoid running compactor concurrently with verify-repair on same bucket","Retry after transient errors"],"tags":["object-storage","deletion-mark","thanos"],"backgroundTag":"file-write-failed","analyzedSha":"35b8b991177def87ed52dcf10f9b6d87f07282c8","analyzedAt":"2026-09-07T01:49:59.689Z","contentChangedAt":"2026-09-07T01:49:59.689Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}