thanos-io/thanos · error

retention failed

Error message

retention failed

What it means

Wrapper error in runCompact (cmd/thanos/compact.go:523) around compact.ApplyRetentionPolicyByResolution. Marking expired blocks for deletion failed; the inner error identifies which block or store operation broke.

Solutions

1. Check the wrapped inner error for the specific failing block/operation
2. Ensure bucket credentials have write permission (deletion marks are written to the bucket)
3. Verify retention flag values are sane and intended (durations with correct units)
4. Retry; retention marking is idempotent for already-marked blocks

Example fix

// before
thanos compact --retention.1h=0s ... # with read-only bucket credentials
// after
# grant write access (PutObject/DeleteObject) to the compactor's service account
thanos compact --retention.1h=336h ...

// before starting compactor, verify write access
iterCtx, cancel := context.WithTimeout(ctx, 30*time.Second)
defer cancel()
// ensure the service account has PutObject/DeleteObject on the bucket
_ = iterCtx

if err := compact.ApplyRetentionPolicyByResolution(...); err != nil {
	level.Error(logger).Log("msg", "retention failed", "err", err)
	return errors.Wrap(err, "retention failed")
}

When it happens

Trigger: ApplyRetentionPolicyByResolution failing while marking blocks past their resolution-specific retention for deletion: object store write/mark failures, meta read errors, or issues with the blocksMarked metric collector.

Common situations: Bucket write permissions missing (read-only credentials can sync but not write deletion marks), retention configs (e.g. retention.1h) deleting many blocks at once hitting store limits, corrupt metas on retention candidates.

Understand the failure class

Background: Permission denied / not authorized / 403 Forbidden: access-control rejections when the caller lacks the required role, grant, or ownership — this error's family across 18 libraries.

Related errors

• sync before retention
• retention failed
• delete block
• raw resolution must be higher than the minimum block size…
• 5m resolution retention must be higher than the minimum…

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/8bb69a51494c993f. Report an issue: GitHub.

Appendix: source

Thrown at cmd/thanos/compact.go:523

				conf.blockFilesConcurrency,
				metadata.HashFunc(conf.hashFunc),
				conf.acceptMalformedIndex,
			); err != nil {
				return errors.Wrap(err, "second pass of downsampling failed")
			}

			level.Info(logger).Log("msg", "downsampling iterations done")
		} else {
			level.Info(logger).Log("msg", "downsampling was explicitly disabled")
		}

		// TODO(bwplotka): Find a way to avoid syncing if no op was done.
		if err := sy.SyncMetas(ctx); err != nil {
			return errors.Wrap(err, "sync before retention")
		}

		if err := compact.ApplyRetentionPolicyByResolution(ctx, logger, insBkt, sy.Metas(), retentionByResolution, compactMetrics.blocksMarked.WithLabelValues(metadata.DeletionMarkFilename, "")); err != nil {
			return errors.Wrap(err, "retention failed")
		}

		return cleanPartialMarked()
	}

	g.Add(func() error {
		defer runutil.CloseWithLogOnErr(logger, insBkt, "bucket client")

		if !conf.wait {
			return compactMainFn()
		}

		// --wait=true is specified.
		return runutil.Repeat(conf.waitInterval, ctx.Done(), func() error {
			err := compactMainFn()
			if err == nil {
				compactMetrics.iterations.Inc()
				return nil

View on GitHub (pinned to 35b8b99117)