thanos-io/thanos · critical

sync blocks

Error message

sync blocks

What it means

The command calls sy.SyncMetas(ctx) to download and load metadata for all blocks in the bucket. Any failure during this phase (list failures, meta.json download/unmarshal errors, context cancellation) is wrapped as 'sync blocks'. This means the bucket view of block metadata could not be fully built.

Solutions

1. Re-run the command; sync is idempotent and transient network/credential failures often resolve.
2. Check the wrapped inner error: if it mentions a specific block, inspect/repair that block's meta.json or delete the partial upload.
3. Verify credentials/session expiry if the run is long; refresh tokens or use instance profiles.
4. Increase timeouts / ensure the context isn't cancelled (don't interrupt during 'syncing blocks metadata').
5. Use `thanos tools bucket verify` to find corrupted blocks, and `thanos tools bucket mark --mark=deletion-mark` + cleanup for unrecoverable ones.

Example fix

// before: fixing a corrupted block meta found by the inner error
// (bucket: thanos-blocks, block 01ARZ.../meta.json is truncated)
thanos tools bucket mark --objstore.config-file=b.yml --mark=deletion-mark --id=01ARZ3NDEKTSV4RRFFQ69G5FAV --details="corrupted meta.json"
// after
thanos tools bucket compact --objstore.config-file=b.yml  # re-run sync after marking the bad block

// Pre-flight: ensure credentials are fresh and bucket is reachable
if err := checkBucketListable(ctx, bkt); err != nil {
    return err // fail fast before SyncMetas
}

if err := sy.SyncMetas(ctx); err != nil {
    if ctx.Err() != nil {
        return fmt.Errorf("sync cancelled: %w", ctx.Err())
    }
    if isRetryable(errors.Unwrap(err)) {
        return retryWithBackoff(func() error { return sy.SyncMetas(ctx) })
    }
    return errors.Wrap(err, "sync blocks")
}

When it happens

Trigger: Running compact/replicate when SyncMetas fails at cmd/thanos/tools_bucket.go:903 — ListObjects errors, corrupted meta.json in the bucket, fetching errors on individual blocks, or the context being cancelled (Ctrl-C / timeout) mid-sync.

Common situations: Flaky object-store connectivity, expired cloud credentials mid-run, partially uploaded/corrupted meta.json from an interrupted upload, or timeouts on very large buckets exceeding the command deadline.

Understand the failure class

Background: "API request failed": what wrapped HTTP errors from external APIs mean and how to find the real cause — this error's family across 29 libraries.

Related errors

• sync before first pass of downsampling
• sync before second pass of downsampling
• sync before retention
• create meta fetcher
• unable to sync blocks

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/74ece8f3a807236e. Report an issue: GitHub.

Appendix: source

Thrown at cmd/thanos/tools_bucket.go:903

			sy, err = compact.NewMetaSyncer(
				logger,
				reg,
				insBkt,
				cf,
				duplicateBlocksFilter,
				ignoreDeletionMarkFilter,
				stubCounter,
				stubCounter,
				0,
			)
			if err != nil {
				return errors.Wrap(err, "create syncer")
			}
		}

		level.Info(logger).Log("msg", "syncing blocks metadata")
		if err := sy.SyncMetas(ctx); err != nil {
			return errors.Wrap(err, "sync blocks")
		}

		level.Info(logger).Log("msg", "synced blocks done")

		compact.BestEffortCleanAbortedPartialUploads(ctx, logger, sy.Partial(), insBkt, stubCounter, stubCounter, stubCounter, ignoreDeletionMarkFilter.DeletionMarkBlocks())
		if _, err := blocksCleaner.DeleteMarkedBlocks(ctx); err != nil {
			return errors.Wrap(err, "error cleaning blocks")
		}

		level.Info(logger).Log("msg", "cleanup done")
		return nil
	})
}

type tablePrinter func(w io.Writer, t Table) error

func printTable(w io.Writer, t Table) error {
	table := tablewriter.NewWriter(w)

View on GitHub (pinned to 35b8b99117)