{"record":{"id":"6b768ac9c9fe1498","repo":"thanos-io/thanos","slug":"safe-deleting-old-block-s-failed","errorCode":null,"errorMessage":"safe deleting old block %s failed","messagePattern":"safe deleting old block (.+?) failed","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"pkg/verifier/index_issue.go","lineNumber":126,"sourceCode":"\t\tblock.IgnoreIssue347OutsideChunk,\n\t)\n\tif err != nil {\n\t\treturn errors.Wrapf(err, \"repair failed for block %s\", id)\n\t}\n\tlevel.Info(ctx.Logger).Log(\"msg\", \"verifying repaired block\", \"id\", id, \"newID\", resid)\n\n\tif err := block.VerifyIndex(ctx, ctx.Logger, filepath.Join(dir, resid.String(), block.IndexFilename), meta.MinTime, meta.MaxTime); err != nil {\n\t\treturn errors.Wrapf(err, \"repaired block is invalid %s\", resid)\n\t}\n\n\tlevel.Info(ctx.Logger).Log(\"msg\", \"uploading repaired block\", \"newID\", resid)\n\tif err = block.Upload(ctx, ctx.Logger, ctx.Bkt, filepath.Join(dir, resid.String()), metadata.NoneFunc); err != nil {\n\t\treturn errors.Wrapf(err, \"upload of %s failed\", resid)\n\t}\n\n\tlevel.Info(ctx.Logger).Log(\"msg\", \"safe deleting broken block\", \"id\", id, \"issue\")\n\tif err := BackupAndDeleteDownloaded(ctx, filepath.Join(dir, id.String()), id); err != nil {\n\t\treturn errors.Wrapf(err, \"safe deleting old block %s failed\", id)\n\t}\n\n\treturn nil\n}\n\nfunc verifyIndex(ctx Context, id ulid.ULID, dir string, meta *metadata.Meta) (stats block.HealthStats, err error) {\n\tif err := objstore.DownloadFile(ctx, ctx.Logger, ctx.Bkt, path.Join(id.String(), block.IndexFilename), filepath.Join(dir, block.IndexFilename)); err != nil {\n\t\treturn stats, errors.Wrapf(err, \"download index file %s\", path.Join(id.String(), block.IndexFilename))\n\t}\n\n\tstats, err = block.GatherIndexHealthStats(ctx, ctx.Logger, filepath.Join(dir, block.IndexFilename), meta.MinTime, meta.MaxTime)\n\tif err != nil {\n\t\treturn stats, errors.Wrapf(err, \"gather index issues %s\", id)\n\t}\n\n\tlevel.Debug(ctx.Logger).Log(\"stats\", fmt.Sprintf(\"%+v\", stats), \"id\", id)\n\n\treturn stats, stats.AnyErr()","sourceCodeStart":108,"sourceCodeEnd":144,"githubUrl":"https://github.com/thanos-io/thanos/blob/35b8b991177def87ed52dcf10f9b6d87f07282c8/pkg/verifier/index_issue.go#L108-L144","documentation":"The last step of repairIndex deletes the broken block from the source bucket after backing it up, via BackupAndDeleteDownloaded; failure is wrapped as \"safe deleting old block %s failed\". The repaired block was uploaded, but the broken original could not be safely removed.","triggerScenarios":"BackupAndDeleteDownloaded fails: the block already exists in the backup bucket, backup upload fails, or the source bucket delete call fails (permission denied, network error).","commonSituations":"Backup bucket containing a previous backup of the same block ID (previous verify run interrupted); read-only or delete-denied IAM policy on the source bucket; transient object store outage during delete.","solutions":["Check the backup bucket: if the block dir already exists there, remove it manually after confirming it is a safe copy, then re-run.","Verify the source bucket credentials allow DeleteObject.","Retry the verify/repair run once the object store is reachable again.","Manually delete the broken block from the source bucket (after backing it up yourself) if the automated path keeps failing."],"exampleFix":"// before: backup bucket shared with other runs holding the same block id\n--backup-bucket=shared-backup\n// after: dedicated backup bucket for this verifier run\n--backup-bucket=thanos-verifier-backup-prod-compact","handlingStrategy":"validation","validationCode":"found, err := verifier.TSDBBlockExistsInBucket(ctx, backupBkt, id)\nif err != nil || found {\n    return fmt.Errorf(\"backup bucket already has %s; clean it first\", id)\n}","typeGuard":null,"tryCatchPattern":"if err := verifier.VerifyRepair(...); err != nil {\n    if strings.Contains(err.Error(), \"safe deleting old block\") {\n        // check source-bucket delete permissions and backup bucket collisions\n    }\n}","preventionTips":["Grant DeleteObject on the source bucket to the verifier identity","Use a dedicated backup bucket to avoid stale-copy collisions","Clean the backup bucket of partial blocks from interrupted runs before re-running"],"tags":["go","object-storage","delete","backup","thanos"],"backgroundTag":"file-already-exists","analyzedSha":"35b8b991177def87ed52dcf10f9b6d87f07282c8","analyzedAt":"2026-09-07T01:49:59.689Z","contentChangedAt":"2026-09-07T01:49:59.689Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}