thanos-io/thanos · error

could not sync metas

Error message

could not sync metas

What it means

The periodic compaction-progress worker calls sy.SyncMetas(ctx) to fetch block metadata from the bucket via the MetaFetcher. If syncing fails with an error that is not a RetryError, the iteration is aborted with 'could not sync metas'. It means the compactor could not obtain a consistent view of the blocks in the bucket.

Solutions

1. Inspect the wrapped inner error; if it points at a specific block's meta.json, validate/repair or remove that block.
2. Test bucket access with 'thanos tools bucket ls' to separate auth/network issues from metadata issues.
3. Retry the process; if the error is transient and misclassified, upgrade Thanos so SyncMetas errors are retried.
4. Alert on frequent occurrences — the whole progress-calculation loop is skipped, so metrics will go stale.

Example fix

// before
// corrupted meta.json causes sync failure every iteration
// after
thanos tools bucket inspect --objstore.bucket=thanos
# or delete/repair the offending block's meta.json, then restart

// Preflight metadata listing before starting the progress worker
metas, _, err := fetcher.Fetch(ctx)
if err != nil {
    log.Printf("metadata preflight failed: %v", err)
}

if err := sy.SyncMetas(ctx); err != nil {
    if compact.IsRetryError(err) {
        return nil // retry next tick
    }
    log.Errorf("could not sync metas: %v", err) // alert; inspect inner block errors
    return nil
}

When it happens

Trigger: sy.SyncMetas(ctx) returns a non-retryable error inside the progressCalculateInterval repeat loop — e.g. bucket ListObjects consistently failing, unparseable meta.json in a block, or an object-store API error not wrapped as retry.

Common situations: Corrupted or hand-edited meta.json in the bucket causing fetch errors, expired object-store credentials, or an S3/GCS outage that returns errors the fetcher does not classify as retriable.

Understand the failure class

Background: 'Something went wrong' / 'Request failed (500)' / 'HTTP error! status: 404' — what failed HTTP requests actually mean and how to find the real cause — this error's family across 28 libraries.

Related errors

• read meta from
• fetch overlaps
• compaction
• sync before first pass of downsampling
• sync before second pass of downsampling

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/f3f6125fa5d22d6d. Report an issue: GitHub.

Appendix: source

Thrown at cmd/thanos/compact.go:660

				rs := compact.NewRetentionProgressCalculator(reg, retentionByResolution)
				var ds *compact.DownsampleProgressCalculator
				if !conf.disableDownsampling {
					ds = compact.NewDownsampleProgressCalculator(reg)
				}

				return runutil.Repeat(conf.progressCalculateInterval, ctx.Done(), func() error {

					if err := sy.SyncMetas(ctx); err != nil {
						// The RetryError signals that we hit an retriable error (transient error, no connection).
						// You should alert on this being triggered too frequently.
						if compact.IsRetryError(err) {
							level.Error(logger).Log("msg", "retriable error", "err", err)
							compactMetrics.retried.Inc()

							return nil
						}

						return errors.Wrapf(err, "could not sync metas")
					}

					metas := sy.Metas()
					groups, err := grouper.Groups(metas)
					if err != nil {
						return errors.Wrapf(err, "could not group metadata for compaction")
					}

					if err = ps.ProgressCalculate(ctx, groups); err != nil {
						return errors.Wrapf(err, "could not calculate compaction progress")
					}

					retGroups, err := grouper.Groups(metas)
					if err != nil {
						return errors.Wrapf(err, "could not group metadata for retention")
					}

					if err = rs.ProgressCalculate(ctx, retGroups); err != nil {

View on GitHub (pinned to 35b8b99117)