thanos-io/thanos · error

sync block

Error message

sync block

What it means

BucketStore.InitialSync wraps errors from SyncBlocks with "sync block". SyncBlocks iterates the object store bucket, downloads missing block metadata and syncs local state; any bucket access, metadata fetch, or block download failure surfaces wrapped as "sync block: <cause>".

Solutions

1. Run `thanos tools bucket verify` to find corrupt/partially uploaded blocks.
2. Verify object store credentials and IAM permissions (ListBucket, GetObject on the bucket prefix).
3. Check network/proxy connectivity to the object storage endpoint from the pod.
4. Remove or repair blocks with invalid meta.json (marked for no-compact-mark / deletion).

Example fix

// before
err := store.InitialSync(ctx) // opaque "sync block: ..." failure
// after
err := store.InitialSync(ctx)
if err != nil {
    logger.Error("initial block sync failed; check bucket credentials and corrupt blocks", "err", err)
    return err
}

if err := checkBucketAccess(ctx, bkt); err != nil {
    return errors.Wrap(err, "bucket not accessible before InitialSync")
}

err := store.InitialSync(ctx)
if err != nil {
    if isTransientStorageErr(err) {
        return retryWithBackoff(ctx, store.InitialSync)
    }
    return errors.Wrap(err, "initial sync failed permanently")
}

When it happens

Trigger: InitialSync (or background sync) hitting object storage failures: missing bucket permissions (s3:GetObject/ListBucket), network timeouts, invalid bucket credentials, or corrupt meta.json in a block.

Common situations: Store Gateway starting with wrong S3/GCS credentials, IAM policy missing list permissions, bucket region mismatch, object storage outage, or a partially-uploaded (failed) block in the bucket.

Understand the failure class

Background: "API request failed": what wrapped HTTP errors from external APIs mean and how to find the real cause — this error's family across 29 libraries.

Related errors

• sync before first pass of downsampling
• sync before second pass of downsampling
• sync before retention
• unable to sync blocks
• compaction

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/f255e6b2fc0fdacf. Report an issue: GitHub.

Appendix: source

Thrown at pkg/store/bucket.go:798

	// Sync advertise labels.
	s.mtx.Lock()
	s.advLabelSets = make([]labelpb.ZLabelSet, 0, len(s.advLabelSets))
	for _, bs := range s.blockSets {
		s.advLabelSets = append(s.advLabelSets, labelpb.ZLabelSet{Labels: labelpb.ZLabelsFromPromLabels(bs.labels.Copy())})
	}
	sort.Slice(s.advLabelSets, func(i, j int) bool {
		return strings.Compare(s.advLabelSets[i].String(), s.advLabelSets[j].String()) < 0
	})
	s.mtx.Unlock()
	return nil
}

// InitialSync perform blocking sync with extra step at the end to delete locally saved blocks that are no longer
// present in the bucket. The mismatch of these can only happen between restarts, so we can do that only once per startup.
func (s *BucketStore) InitialSync(ctx context.Context) error {
	if err := s.SyncBlocks(ctx); err != nil {
		return errors.Wrap(err, "sync block")
	}

	if s.dir == "" {
		return nil
	}

	fis, err := os.ReadDir(s.dir)
	if err != nil {
		return errors.Wrap(err, "read dir")
	}
	names := make([]string, 0, len(fis))
	for _, fi := range fis {
		names = append(names, fi.Name())
	}
	for _, n := range names {
		id, ok := block.IsBlockDir(n)
		if !ok {
			continue

View on GitHub (pinned to 35b8b99117)