thanos-io/thanos · error

BaseFetcher: iter bucket

Error message

BaseFetcher: iter bucket

What it means

fetchMetadata runs errgroup goroutines that iterate the bucket via blockIDsLister.GetActiveAndPartialBlockIDs. If that listing fails (any object-store error during List), eg.Wait() returns it and fetchMetadata wraps it with "BaseFetcher: iter bucket". It is a fatal sync error: the block list itself could not be obtained, so no metadata fetch proceeds.

Solutions

1. Read the wrapped inner error to see the object-store cause; verify credentials and List permission on the bucket (e.g. s3:ListBucket).
2. Verify bucket name, region/endpoint in the bucket client configuration.
3. Test bucket connectivity with the provider CLI (aws s3 ls / gsutil ls) from the same host/network.
4. Retry on transient network errors; check provider status page for outages.

Example fix

// before: bucket config with wrong region/missing permission
// s3: endpoint=s3.amazonaws.com, bucket=my-blocks (us-east-1 bucket, client in eu-west-1)
// after: fix region and grant list permission
// s3: endpoint=s3.eu-west-1.amazonaws.com, bucket=my-blocks, IAM policy allows s3:ListBucket

// Before sync: verify bucket list access
_, err := bucket.Iter(ctx, "", func(string) error { return nil })
if err != nil {
    // fail fast with a clear config/permission message
}

metas, err := fetcher.FetchMetadata(ctx, filtered)
if err != nil && strings.Contains(err.Error(), "BaseFetcher: iter bucket") {
    // transient object-store failure: backoff and retry sync
    time.Sleep(retryBackoff)
    metas, err = fetcher.FetchMetadata(ctx, filtered)
}

When it happens

Trigger: Calling fetchMetadata/FetchMetadata (sync) when the underlying object store List/Iter call fails — expired credentials, network failure, bucket missing, or bucket lister misconfiguration.

Common situations: Wrong bucket name or region in store config; IAM credentials lacking s3:ListBucket (or equivalent) permission; S3/GCS outage or VPC without egress; typos in endpoint configuration.

Understand the failure class

Background: 'Something went wrong' / 'Request failed (500)' / 'HTTP error! status: 404' — what failed HTTP requests actually mean and how to find the real cause — this error's family across 28 libraries.

Related errors

• sync before first pass of downsampling
• sync before second pass of downsampling
• sync before retention
• retention failed
• sync before first pass of downsampling

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/445703f7e9b3ca7e. Report an issue: GitHub.

Appendix: source

Thrown at pkg/block/fetcher.go:593

				mtx.Lock()
				resp.partial[id] = err
				mtx.Unlock()
			}
			return nil
		})
	}

	var partialBlocks map[ulid.ULID]bool
	var err error
	// Workers scheduled, distribute blocks.
	eg.Go(func() error {
		defer close(activeBlocksCh)
		partialBlocks, err = f.blockIDsLister.GetActiveAndPartialBlockIDs(ctx, activeBlocksCh)
		return err
	})

	if err := eg.Wait(); err != nil {
		return nil, errors.Wrap(err, "BaseFetcher: iter bucket")
	}

	mtx.Lock()
	for blockULID, isPartial := range partialBlocks {
		if isPartial {
			resp.partial[blockULID] = errors.Errorf("block %s has no meta file", blockULID)
			resp.noMetas++
		}
	}
	mtx.Unlock()

	if len(resp.metaErrs) > 0 {
		return resp, nil
	}

	// Only for complete view of blocks update the cache.

	cached := &sync.Map{}

View on GitHub (pinned to 35b8b99117)