thanos-io/thanos · error

delete from source

Error message

delete from source

What it means

BackupAndDelete moves a TSDB block to a backup bucket and then removes it from the source bucket. This error wraps a failure from block.Delete when deleting the block from the source bucket (only attempted when ctx.DeleteDelay is zero). It indicates the block was backed up but could not be removed from the origin bucket.

Solutions

1. Check bucket credentials/IAM allow delete operations on the source bucket
2. Check for object-lock/retention or versioning policies preventing deletion
3. Re-run the verify-repair; the block is already backed up so the delete can be retried
4. Inspect the wrapped cause in the error chain for the underlying object-store error

Example fix

// before
if ctx.DeleteDelay.Seconds() == 0 { ... block.Delete ... }
// after
if err := block.Delete(ctx, ctx.Logger, ctx.Bkt, id); err != nil {
	level.Warn(ctx.Logger).Log("msg", "failed to delete block from source; will be marked for deletion", "id", id.String(), "err", err)
	return errors.Wrap(err, "delete from source")
}

// Go: pre-check delete permission if possible
allowed, err := sourceBkt.Iter(ctx, "", func(string) error { return nil })
if err != nil { /* bucket unreadable; expect delete problems too */ }
_ = allowed

if err := runVerifyRepair(); err != nil {
	if strings.Contains(err.Error(), "delete from source") {
		// backup succeeded; retry later or delete manually
	}
}

When it happens

Trigger: Calling VerifyRepair which invokes BackupAndDelete with ctx.DeleteDelay == 0, after the block was successfully uploaded to the backup bucket, and block.Delete(ctx, ctx.Logger, ctx.Bkt, id) fails (e.g. bucket permission errors, object lock/retention, transient storage errors).

Common situations: Running thanos verify with --delete-delay=0 against a bucket where the credentials lack delete permission; S3/GCS object versioning or retention policies blocking deletes; transient object store outages during verify-repair.

Related errors

• compaction
• sync before first pass of downsampling
• sync before second pass of downsampling
• sync before retention
• retention failed

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/bf7586ff3b0a2545. Report an issue: GitHub.

Appendix: source

Thrown at pkg/verifier/safe_delete.go:79

		}
	}()

	// Download the TSDB block.
	dir := filepath.Join(tempdir, id.String())
	if err := block.Download(ctx, ctx.Logger, ctx.Bkt, id, dir); err != nil {
		return errors.Wrap(err, "download from source")
	}

	// Backup the block.
	if err := backupDownloaded(ctx, ctx.Logger, dir, ctx.BackupBkt, id); err != nil {
		return err
	}

	// Block uploaded, so we are ok to remove from src bucket.
	if ctx.DeleteDelay.Seconds() == 0 {
		level.Info(ctx.Logger).Log("msg", "Deleting block", "id", id.String())
		if err := block.Delete(ctx, ctx.Logger, ctx.Bkt, id); err != nil {
			return errors.Wrap(err, "delete from source")
		}
	}

	level.Info(ctx.Logger).Log("msg", "Marking block as deleted", "id", id.String())
	if err := block.MarkForDeletion(ctx, ctx.Logger, ctx.Bkt, id, "manual verify-repair", ctx.metrics.blocksMarkedForDeletion); err != nil {
		return errors.Wrap(err, "marking delete from source")
	}
	return nil
}

// BackupAndDeleteDownloaded works much like BackupAndDelete in that it will
// move a TSDB block from a bucket to a backup bucket. If deleteDelay param is zero, block is removed from source bucket.
// else the block is marked for deletion. The bdir parameter
// points to the location on disk where the TSDB block was previously
// downloaded allowing this function to avoid downloading the TSDB block from
// the source bucket again. An error is returned if any operation fails.
func BackupAndDeleteDownloaded(ctx Context, bdir string, id ulid.ULID) error {
	// Does this TSDB block exist in backupBkt already?

View on GitHub (pinned to 35b8b99117)