thanos-io/thanos · error

marking delete from source

Error message

marking delete from source

What it means

After backing up a block, BackupAndDelete marks it for deletion in the source bucket via block.MarkForDeletion when DeleteDelay is non-zero. This error wraps a failure of that mark operation, meaning the deletion marker (deletion-mark.json) could not be written to the source bucket.

Solutions

1. Grant write permission to the source bucket for deletion-mark.json objects
2. Ensure no conflicting compactor process is overriding the deletion mark
3. Retry verify-repair; block is safely backed up already
4. Check the wrapped error for the underlying object-store cause

// ensure write access by touching a test object
err := markBkt.Upload(ctx, "__perm_test__", bytes.NewReader(nil))

if err := runVerifyRepair(); err != nil {
	if strings.Contains(err.Error(), "marking delete from source") {
		// block backed up; fix bucket write perms and retry
	}
}

When it happens

Trigger: BackupAndDelete invoked with ctx.DeleteDelay > 0; the block is backed up but block.MarkForDeletion(ctx, ctx.Logger, ctx.Bkt, id, "manual verify-repair", ...) returns an error (write failure to source bucket).

Common situations: Read-only or partially-scoped bucket credentials missing write permission for deletion-mark.json; compactor concurrently holding/overwriting the mark; transient bucket errors during verify --repair runs.

Understand the failure class

Background: "failed to write file", "Could not save figure", "Error saving remote file" — file write failed: causes and fixes across languages and libraries — this error's family across 38 libraries.

Related errors

• compaction
• sync before first pass of downsampling
• sync before second pass of downsampling
• sync before retention
• retention failed

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/d0b3096d1e75b10d. Report an issue: GitHub.

Appendix: source

Thrown at pkg/verifier/safe_delete.go:85

		return errors.Wrap(err, "download from source")
	}

	// Backup the block.
	if err := backupDownloaded(ctx, ctx.Logger, dir, ctx.BackupBkt, id); err != nil {
		return err
	}

	// Block uploaded, so we are ok to remove from src bucket.
	if ctx.DeleteDelay.Seconds() == 0 {
		level.Info(ctx.Logger).Log("msg", "Deleting block", "id", id.String())
		if err := block.Delete(ctx, ctx.Logger, ctx.Bkt, id); err != nil {
			return errors.Wrap(err, "delete from source")
		}
	}

	level.Info(ctx.Logger).Log("msg", "Marking block as deleted", "id", id.String())
	if err := block.MarkForDeletion(ctx, ctx.Logger, ctx.Bkt, id, "manual verify-repair", ctx.metrics.blocksMarkedForDeletion); err != nil {
		return errors.Wrap(err, "marking delete from source")
	}
	return nil
}

// BackupAndDeleteDownloaded works much like BackupAndDelete in that it will
// move a TSDB block from a bucket to a backup bucket. If deleteDelay param is zero, block is removed from source bucket.
// else the block is marked for deletion. The bdir parameter
// points to the location on disk where the TSDB block was previously
// downloaded allowing this function to avoid downloading the TSDB block from
// the source bucket again. An error is returned if any operation fails.
func BackupAndDeleteDownloaded(ctx Context, bdir string, id ulid.ULID) error {
	// Does this TSDB block exist in backupBkt already?
	found, err := TSDBBlockExistsInBucket(ctx, ctx.BackupBkt, id)
	if err != nil {
		return err
	}
	if found {
		return errors.Errorf("%s dir seems to exists in backup bucket. Remove this block manually if you are sure it is safe to do", id)

View on GitHub (pinned to 35b8b99117)