thanos-io/thanos · error

safe deleting old block

Error message

safe deleting old block %s failed

What it means

The last step of repairIndex deletes the broken block from the source bucket after backing it up, via BackupAndDeleteDownloaded; failure is wrapped as "safe deleting old block %s failed". The repaired block was uploaded, but the broken original could not be safely removed.

Solutions

1. Check the backup bucket: if the block dir already exists there, remove it manually after confirming it is a safe copy, then re-run.
2. Verify the source bucket credentials allow DeleteObject.
3. Retry the verify/repair run once the object store is reachable again.
4. Manually delete the broken block from the source bucket (after backing it up yourself) if the automated path keeps failing.

Example fix

// before: backup bucket shared with other runs holding the same block id
--backup-bucket=shared-backup
// after: dedicated backup bucket for this verifier run
--backup-bucket=thanos-verifier-backup-prod-compact

found, err := verifier.TSDBBlockExistsInBucket(ctx, backupBkt, id)
if err != nil || found {
    return fmt.Errorf("backup bucket already has %s; clean it first", id)
}

if err := verifier.VerifyRepair(...); err != nil {
    if strings.Contains(err.Error(), "safe deleting old block") {
        // check source-bucket delete permissions and backup bucket collisions
    }
}

When it happens

Trigger: BackupAndDeleteDownloaded fails: the block already exists in the backup bucket, backup upload fails, or the source bucket delete call fails (permission denied, network error).

Common situations: Backup bucket containing a previous backup of the same block ID (previous verify run interrupted); read-only or delete-denied IAM policy on the source bucket; transient object store outage during delete.

Understand the failure class

Background: "already exists" / EEXIST / FileAlreadyExistsException: what the 'file already exists' error means and how to fix it — this error's family across 37 libraries.

Related errors

• dir seems to exists in backup bucket. Remove this block…
• unable to create bucket
• unable to sync blocks
• delete block
• upload of failed

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/6b768ac9c9fe1498. Report an issue: GitHub.

Appendix: source

Thrown at pkg/verifier/index_issue.go:126

		block.IgnoreIssue347OutsideChunk,
	)
	if err != nil {
		return errors.Wrapf(err, "repair failed for block %s", id)
	}
	level.Info(ctx.Logger).Log("msg", "verifying repaired block", "id", id, "newID", resid)

	if err := block.VerifyIndex(ctx, ctx.Logger, filepath.Join(dir, resid.String(), block.IndexFilename), meta.MinTime, meta.MaxTime); err != nil {
		return errors.Wrapf(err, "repaired block is invalid %s", resid)
	}

	level.Info(ctx.Logger).Log("msg", "uploading repaired block", "newID", resid)
	if err = block.Upload(ctx, ctx.Logger, ctx.Bkt, filepath.Join(dir, resid.String()), metadata.NoneFunc); err != nil {
		return errors.Wrapf(err, "upload of %s failed", resid)
	}

	level.Info(ctx.Logger).Log("msg", "safe deleting broken block", "id", id, "issue")
	if err := BackupAndDeleteDownloaded(ctx, filepath.Join(dir, id.String()), id); err != nil {
		return errors.Wrapf(err, "safe deleting old block %s failed", id)
	}

	return nil
}

func verifyIndex(ctx Context, id ulid.ULID, dir string, meta *metadata.Meta) (stats block.HealthStats, err error) {
	if err := objstore.DownloadFile(ctx, ctx.Logger, ctx.Bkt, path.Join(id.String(), block.IndexFilename), filepath.Join(dir, block.IndexFilename)); err != nil {
		return stats, errors.Wrapf(err, "download index file %s", path.Join(id.String(), block.IndexFilename))
	}

	stats, err = block.GatherIndexHealthStats(ctx, ctx.Logger, filepath.Join(dir, block.IndexFilename), meta.MinTime, meta.MaxTime)
	if err != nil {
		return stats, errors.Wrapf(err, "gather index issues %s", id)
	}

	level.Debug(ctx.Logger).Log("stats", fmt.Sprintf("%+v", stats), "id", id)

	return stats, stats.AnyErr()

View on GitHub (pinned to 35b8b99117)