thanos-io/thanos · error

marking old block for deletion has failed

Error message

marking old block %s for deletion has failed

What it means

This error wraps a failure of block.MarkForDeletion in the final step of repairBucketBlock (compact.go). After the repaired replacement block was uploaded, the compactor marks the original broken block (ie.id) in the bucket with a deletion marker so it will be garbage-collected. If marking fails, the bucket is left with both the broken block and the repaired one, which can later cause overlap-halt of compaction (as the TODO in the source notes).

Solutions

1. Manually mark or delete the broken block: create the deletion-mark.json object for ie.id or run 'thanos tools bucket verify'/'bucket mark --marker=deletion-mark' so the overlap does not halt compaction
2. Fix bucket write permissions for the compactor (it needs write access for markers, not just block upload)
3. Ensure a single compactor instance/group per bucket prefix to avoid concurrent repair/marker races
4. If storage latency is the cause, the compactor retries the whole repair cycle; still clean the duplicate manually to avoid the overlap-halt described in the TODO
5. Check object storage availability/5xx errors at the time of failure and re-run compaction

Example fix

// before: broken block left in bucket, compaction halts on overlap
// after: mark it for deletion manually
$ thanos tools bucket mark --marker=deletion-mark \
    --objstore.config-file=bucket.yaml --id=<brokenBlockULID> \
    --details="source of repaired block"

// ensure write access for deletion markers before starting the compactor
if err := bkt.Upload(ctx, "marker-probe", strings.NewReader("probe")); err != nil {
    return fmt.Errorf("cannot write deletion markers: %w", err)
}

if err := block.MarkForDeletion(delCtx, logger, bkt, ie.id, "source of repaired block", blocksMarkedForDeletion); err != nil {
    logger.Error("failed to mark repaired-over block; clean it manually to avoid overlap halt",
        "id", ie.id, "err", err)
    return errors.Wrapf(err, "marking old block %s for deletion has failed", ie.id)
}

When it happens

Trigger: block.MarkForDeletion(delCtx, logger, bkt, ie.id, "source of repaired block", blocksMarkedForDeletion) fails: object storage write of the deletion marker fails (permission, network, 5xx), the 5-minute delCtx timeout expires on a slow storage backend, or the block was already marked/deleted concurrently by another compactor.

Common situations: Slow or rate-limited object storage making the marker PUT exceed the 5m timeout; credentials allowing read but not write of marker objects; two compactor instances repairing the same overlap simultaneously; transient network blip exactly at the cleanup step after a long repair.

Understand the failure class

Background: "failed to write file", "Could not save figure", "Error saving remote file" — file write failed: causes and fixes across languages and libraries — this error's family across 38 libraries.

Related errors

• read meta from
• compaction
• sync before first pass of downsampling
• sync before second pass of downsampling
• sync before retention

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/ae20fd6a1582803e. Report an issue: GitHub.

Appendix: source

Thrown at pkg/compact/compact.go:1162

	// Verify repaired id before uploading it.
	if err := block.VerifyIndex(ctx, logger, filepath.Join(tmpdir, resid.String(), block.IndexFilename), meta.MinTime, meta.MaxTime); err != nil {
		return errors.Wrapf(err, "repaired block is invalid %s", resid)
	}

	level.Info(logger).Log("msg", "uploading repaired block", "newID", resid)
	if err = block.Upload(ctx, logger, bkt, filepath.Join(tmpdir, resid.String()), metadata.NoneFunc); err != nil {
		return retry(errors.Wrapf(err, "upload of %s failed", resid))
	}

	level.Info(logger).Log("msg", "deleting broken block", "id", ie.id)

	// Spawn a new context so we always mark a block for deletion in full on shutdown.
	delCtx, cancel := context.WithTimeout(context.Background(), 5*time.Minute)
	defer cancel()

	// TODO(bplotka): Issue with this will introduce overlap that will halt compactor. Automate that (fix duplicate overlaps caused by this).
	if err := block.MarkForDeletion(delCtx, logger, bkt, ie.id, "source of repaired block", blocksMarkedForDeletion); err != nil {
		return errors.Wrapf(err, "marking old block %s for deletion has failed", ie.id)
	}
	return nil
}

func (cg *Group) compact(ctx context.Context, dir string, planner Planner, comp Compactor, blockDeletableChecker BlockDeletableChecker, compactionLifecycleCallback CompactionLifecycleCallback, errChan chan error) (bool, []ulid.ULID, error) {
	cg.mtx.Lock()
	defer cg.mtx.Unlock()

	// Check for overlapped blocks.
	overlappingBlocks := false
	if err := cg.areBlocksOverlapping(nil); err != nil {
		// TODO(bwplotka): It would really nice if we could still check for other overlaps than replica. In fact this should be checked
		// in syncer itself. Otherwise with vertical compaction enabled we will sacrifice this important check.
		if !cg.enableVerticalCompaction {
			return false, nil, halt(errors.Wrap(err, "pre compaction overlap check"))
		}

		overlappingBlocks = true

View on GitHub (pinned to 35b8b99117)