thanos-io/thanos · error

unable to sync blocks

Error message

unable to sync blocks

What it means

This error is returned when shipper.Sync(ctx) fails during the upload command's run group. Sync walks the local directory, uploads new blocks to the bucket, and updates the meta; any failure reading local blocks, uploading to object storage, or updating remote metadata surfaces here wrapped as 'unable to sync blocks'. The count of uploaded blocks is only logged on success.

Solutions

1. Inspect the wrapped underlying error: if it mentions object storage auth, refresh credentials (env vars, credential files, instance roles)
2. Verify the bucket exists and is writable with the configured credentials (thanos tools bucket verify)
3. Look for corrupted local blocks (invalid meta.json / missing files) in the path and remove or repair them
4. Check network connectivity/proxy settings to the object-store endpoint and retry; if a concurrent compaction is writing blocks, run the upload against a quiesced directory

// Before running sync, verify bucket reachability and credentials:
// thanos tools bucket verify --objstore.bucket.config-file=bucket.yaml
// and locally: find <path> -name meta.json -exec jq -e . {} \; to catch corrupt blocks

n, err := s.Sync(ctx)
if err != nil {
    if ctx.Err() != nil {
        return ctx.Err() // cancelled; do not retry
    }
    logger.Error(err, "block sync failed; check object-store credentials/network and local block integrity")
    return backoff.Retry(func() error { _, err := s.Sync(ctx); return err }, backoff.NewExponentialBackOff())
}

When it happens

Trigger: Underlying object-store operations fail (auth rejected, bucket missing, network timeout, rate limiting), a local block is corrupted or partially written (bad meta.json, missing chunks), or the context is cancelled mid-sync.

Common situations: Expired or missing cloud credentials (AWS/GCS); bucket name wrong or bucket deleted; network/proxy issues blocking storage endpoints; partially-downloaded or corrupted local TSDB blocks; uploading blocks whose external labels conflict or compaction is in progress concurrently.

Understand the failure class

Background: "API request failed": what wrapped HTTP errors from external APIs mean and how to find the real cause — this error's family across 29 libraries.

Related errors

• sync before first pass of downsampling
• sync before second pass of downsampling
• sync before retention
• unable to create bucket
• delete block

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/29c202e789b82d32. Report an issue: GitHub.

Appendix: source

Thrown at cmd/thanos/tools_bucket.go:1513

			bkt,
			tbcDir,
			shipper.WithLogger(logger),
			shipper.WithRegisterer(reg),
			shipper.WithSource(metadata.BucketUploadSource),
			shipper.WithMetaFileName(shipper.DefaultMetaFilename),
			shipper.WithLabels(func() labels.Labels { return lset }),
			shipper.WithUploadCompacted(tbc.uploadCompacted),
			shipper.WithUploadConcurrency(tbc.uploadConcurrency),
		)

		ctx, cancel := context.WithCancel(context.Background())
		g.Add(func() error {
			defer runutil.CloseWithLogOnErr(logger, bkt, "bucket client")
			defer runutil.CloseWithLogOnErr(logger, tbcDir, "tbc directory")

			n, err := s.Sync(ctx)
			if err != nil {
				return errors.Wrap(err, "unable to sync blocks")
			}
			level.Info(logger).Log("msg", "synced blocks", "uploaded", n)
			return nil
		}, func(error) {
			cancel()
		})

		return nil
	})
}

View on GitHub (pinned to 35b8b99117)