thanos-io/thanos · error

sync before retention

Error message

sync before retention

What it means

Wrapper error in runCompact (cmd/thanos/compact.go:519) around sy.SyncMetas(ctx) before applying retention. Retention decisions need an up-to-date view of bucket metas; the sync failure aborts the compaction cycle.

Solutions

1. Check the wrapped inner error for the failing store call
2. Retry the run — retention will re-run on next cycle
3. Validate credentials/TTL for long-running processes
4. Add retry/backoff and timeouts to the bucket client configuration
5. Monitor object-store error metrics correlated with compaction runs

if err := sy.SyncMetas(ctx); err != nil {
	if errors.Is(err, context.Canceled) { return err }
	return errors.Wrap(err, "sync before retention")
}

When it happens

Trigger: sy.SyncMetas(ctx) error after compaction/downsampling: object store list/get failures, throttling, network interruption, corrupt meta objects.

Common situations: Long-running compaction cycles exceeding object-store credential lifetimes; transient bucket outages; rate limiting on very large buckets.

Understand the failure class

Background: 'Something went wrong' / 'Request failed (500)' / 'HTTP error! status: 404' — what failed HTTP requests actually mean and how to find the real cause — this error's family across 28 libraries.

Related errors

• sync before first pass of downsampling
• sync before second pass of downsampling
• retention failed
• unable to sync blocks
• delete block

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/4ebbf45c72120167. Report an issue: GitHub.

Appendix: source

Thrown at cmd/thanos/compact.go:519

				insBkt,
				filteredMetas,
				downsamplingDir,
				conf.downsampleConcurrency,
				conf.blockFilesConcurrency,
				metadata.HashFunc(conf.hashFunc),
				conf.acceptMalformedIndex,
			); err != nil {
				return errors.Wrap(err, "second pass of downsampling failed")
			}

			level.Info(logger).Log("msg", "downsampling iterations done")
		} else {
			level.Info(logger).Log("msg", "downsampling was explicitly disabled")
		}

		// TODO(bwplotka): Find a way to avoid syncing if no op was done.
		if err := sy.SyncMetas(ctx); err != nil {
			return errors.Wrap(err, "sync before retention")
		}

		if err := compact.ApplyRetentionPolicyByResolution(ctx, logger, insBkt, sy.Metas(), retentionByResolution, compactMetrics.blocksMarked.WithLabelValues(metadata.DeletionMarkFilename, "")); err != nil {
			return errors.Wrap(err, "retention failed")
		}

		return cleanPartialMarked()
	}

	g.Add(func() error {
		defer runutil.CloseWithLogOnErr(logger, insBkt, "bucket client")

		if !conf.wait {
			return compactMainFn()
		}

		// --wait=true is specified.
		return runutil.Repeat(conf.waitInterval, ctx.Done(), func() error {

View on GitHub (pinned to 35b8b99117)