thanos-io/thanos · error

delete block

Error message

delete block

What it means

ApplyRetentionPolicyByResolution wraps any failure from block.MarkForDeletion — the operation that writes a deletion marker (meta.json update/marker object) into object storage for blocks older than the retention period. The wrap means the block was identified as exceeding retention, but persisting its deletion mark failed, so retention was not applied for that block.

Solutions

  1. Check object storage connectivity and credentials (bucket write permissions); retry the compactor — MarkForDeletion is retried safely on the next retention run.
  2. Inspect the wrapped cause (errors.Cause) to see the bucket-level error (e.g. NoSuchKey, AccessDenied) and fix storage-side.
  3. Verify the block's meta.json is readable/corrupt-free; delete or repair broken blocks manually if the store returns decode errors.
  4. Reduce retention-scan pressure: ensure the compactor isn't racing a concurrent deletion from another instance (use a single retention owner or blocking).

Example fix

// before: retention loop aborts on first storage error
if err := block.MarkForDeletion(ctx, logger, bkt, id, msg, marked); err != nil {
    return errors.Wrap(err, "delete block")
}
// after: log and continue, letting the next run retry
if err := block.MarkForDeletion(ctx, logger, bkt, id, msg, marked); err != nil {
    level.Error(logger).Log("msg", "failed to mark block for deletion; will retry next run", "id", id, "err", err)
    continue
}
Defensive patterns

Strategy: retry

Validate before calling

// verify bucket write access before running retention
const maxTime = m.MaxTime / 1000
if Date.now() / 1000 <= maxTime + retentionSeconds {
    return null // block within retention, skip MarkForDeletion
}
// pre-flight: attempt a no-op object write to validate credentials/permissions

Try / catch

for attempt := 0; attempt < 3; attempt++ {
    err := ApplyRetentionPolicyByResolution(ctx, logger, bkt, cfg)
    if err == nil { break }
    if !isTransientStorageErr(errors.Cause(err)) { return err }
    time.Sleep(backoff(attempt))
}

Prevention

When it happens

Trigger: During retention compaction (vertical/horizontal sync of blocks per resolution): maxTime = MaxTime/1000 is in the past beyond retentionDuration, and block.MarkForDeletion fails when writing the deletion marker to the bucket (bkt).

Common situations: Object store outages or throttling (S3/GCS 5xx, rate limits), expired/insufficient bucket credentials lacking write permission, corrupt block meta.json in the store, network partitions between Thanos compactor and object storage.

Understand the failure class

Background: "failed to write file", "Could not save figure", "Error saving remote file" — file write failed: causes and fixes across languages and libraries — this error's family across 38 libraries.

Related errors

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/d757b65a210da6cd. Report an issue: GitHub.

Appendix: source

Thrown at pkg/compact/retention.go:44

	ctx context.Context,
	logger log.Logger,
	bkt objstore.Bucket,
	metas map[ulid.ULID]*metadata.Meta,
	retentionByResolution map[ResolutionLevel]time.Duration,
	blocksMarkedForDeletion prometheus.Counter,
) error {
	level.Info(logger).Log("msg", "start optional retention")
	for id, m := range metas {
		retentionDuration := retentionByResolution[ResolutionLevel(m.Thanos.Downsample.Resolution)]
		if retentionDuration.Seconds() == 0 {
			continue
		}

		maxTime := time.Unix(m.MaxTime/1000, 0)
		if time.Now().After(maxTime.Add(retentionDuration)) {
			level.Info(logger).Log("msg", "applying retention: marking block for deletion", "id", id, "maxTime", maxTime.String())
			if err := block.MarkForDeletion(ctx, logger, bkt, id, fmt.Sprintf("block exceeding retention of %v", retentionDuration), blocksMarkedForDeletion); err != nil {
				return errors.Wrap(err, "delete block")
			}
		}
	}
	level.Info(logger).Log("msg", "optional retention apply done")
	return nil
}

View on GitHub (pinned to 35b8b99117)