thanos-io/thanos · error

retention failed

Error message

retention failed

What it means

Returned when compact.ApplyRetentionPolicyByResolution fails after the metadata sync. This step deletes blocks exceeding the retention configured per resolution (e.g. --retention-resolution-raw). Any failure during the retention delete pass is wrapped with 'retention failed'.

Solutions

1. Read the wrapped root cause; most often it is a delete-permission error on the bucket.
2. Grant delete permissions to the credentials used by the tool.
3. Re-run the tool after fixing storage access; retention is idempotent.
4. If block-marking fails due to concurrent compaction, stop the global compactor before running retention.

// Verify delete permission before running retention
bucket, err := objstore.NewClient(logger, cfg)
if err != nil { return err }
iterCtx, cancel := context.WithTimeout(ctx, 30*time.Second)
defer cancel()
err = bucket.Iter(iterCtx, "", func(string) error { return nil })
if err != nil { return fmt.Errorf("bucket not listable: %w", err) }
// For S3 also assert s3:DeleteObject in the IAM policy out-of-band.

if err := compact.ApplyRetentionPolicyByResolution(ctx, logger, bkt, metas, retention, deleter); err != nil {
    log.Printf("retention failed (check delete permissions): %v", err)
    // retention is idempotent; safe to re-run after fixing access
}

When it happens

Trigger: Running a bucket tools command with retention flags set, when ApplyRetentionPolicyByResolution cannot mark/delete expired blocks — e.g. delete permission denied on the bucket, or failure re-syncing metas to decide deletions.

Common situations: Object storage credentials lack delete permissions (S3 s3:DeleteObject), read-only bucket policies, or object store errors while marking blocks for deletion.

Understand the failure class

Background: "You do not have permission" / 403 Forbidden errors: authenticated but not allowed — causes and fixes across open-source libraries — this error's family across 31 libraries.

Related errors

• retention failed
• sync before retention
• delete
• BaseFetcher: iter bucket
• delete block

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/f00e64a905d27f59. Report an issue: GitHub.

Appendix: source

Thrown at cmd/thanos/tools_bucket.go:1451

				0,
			)
			if err != nil {
				return errors.Wrap(err, "create syncer")
			}
		}

		ctx := context.Background()
		level.Info(logger).Log("msg", "syncing blocks metadata")
		if err := sy.SyncMetas(ctx); err != nil {
			return errors.Wrap(err, "sync blocks")
		}

		level.Info(logger).Log("msg", "synced blocks done")

		level.Warn(logger).Log("msg", "GLOBAL COMPACTOR SHOULD __NOT__ BE RUNNING ON THE SAME BUCKET")

		if err := compact.ApplyRetentionPolicyByResolution(ctx, logger, insBkt, sy.Metas(), retentionByResolution, stubCounter); err != nil {
			return errors.Wrap(err, "retention failed")
		}
		return nil
	})
}

func registerBucketUploadBlocks(app extkingpin.AppClause, objStoreConfig *extflag.PathOrContent) {
	cmd := app.Command("upload-blocks", "Upload blocks push blocks from the provided path to the object storage.")

	tbc := &bucketUploadBlocksConfig{}
	tbc.registerBucketUploadBlocksFlag(cmd)

	cmd.Setup(func(g *run.Group, logger log.Logger, reg *prometheus.Registry, _ opentracing.Tracer, _ <-chan struct{}, _ bool) error {
		if len(tbc.labels) == 0 {
			return errors.New("no external labels configured, uniquely identifying external labels must be configured; see https://thanos.io/tip/thanos/storage.md#external-labels for details.")
		}

		lset, err := parseFlagLabels(tbc.labels)
		if err != nil {

View on GitHub (pinned to 35b8b99117)