thanos-io/thanos · error

fetch postings for block

Error message

fetch postings for block %s

What it means

BucketStore's Series RPC fetches matching postings (series refs) from each TSDB block via a remote store client (ExpandPostings) and wraps any failure with the block's ULID. It indicates the underlying block client (a StoreAPI gRPC call) failed or was cancelled while resolving label matchers to series for that specific block.

Solutions

1. Retry the query; transient gRPC errors to one block are often resolved by re-querying after store gateway refreshes block metadata.
2. Check connectivity and logs of the remote store/sidecar serving block blk.meta.ULID (the ULID in the message identifies it).
3. Verify the block still exists in the bucket and is not pending deletion (compaction); remove stale blocks from the bucket or restart store gateway to refresh its view.
4. If timeouts recur on large blocks, increase query timeout, enable response compression, or tune posting offsets in-memory cache settings.

Example fix

// before: no per-block resilience, one bad block fails the query
resp, err := query(ctx, matchers)
// after: rely on partial-response / retry at query layer
q, err := querier.New(...).WithSeriesSelector(matchers)... 
// or wrap with retry:
if errors.Is(err, context.DeadlineExceeded) { err = retryQuery(ctx, matchers) }

// Verify block exists and matches before querying:
meta, err := bucket.Get(ctx, userBucket, path.Join(blkULID.String(), "meta.json"))
if err != nil { return err } // block unavailable; skip or resync

if err != nil {
    var st status.Status
    if errors.As(err, &st) && st.Code() == codes.Unavailable {
        // backoff and retry the query
    }
}

When it happens

Trigger: Calling Series on a BucketStore/store gateway when the per-block ExpandPostings gRPC call to a remote store (or the block's index processing) fails: network drop, remote store shutdown, context deadline exceeded during query, or corrupt index preventing postings expansion.

Common situations: Queries hitting stale block clients after blocks were deleted/compacted away; store-gateway to store-sidecar network issues; query cancellation/timeouts on large blocks; the seriesLimiter rejecting the request is surfaced through this wrap path too.

Understand the failure class

Background: "API request failed": what wrapped HTTP errors from external APIs mean and how to find the real cause — this error's family across 29 libraries.

Related errors

• iterate series for block
• no query API server reachable
• failed to get tsdb status from prometheus
• retrieving targets
• error retrieving rules

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/6e48a598b82c30b5. Report an issue: GitHub.

Appendix: source

Thrown at pkg/store/bucket.go:1679

				s.metrics.lazyExpandedPostingSeriesOverfetchedSizeBytes,
				tenant,
			)

			defer blockClient.Close()

			g.Go(func() error {
				onClose := func() {
					mtx.Lock()
					stats = blockClient.MergeStats(stats)
					mtx.Unlock()
				}

				if err := blockClient.ExpandPostings(
					sortedBlockMatchers,
					seriesLimiter,
				); err != nil {
					onClose()
					return errors.Wrapf(err, "fetch postings for block %s", blk.meta.ULID)
				}

				var resp respSet
				if s.sortingStrategy == sortingStrategyStore {
					resp = newEagerRespSet(
						10*time.Minute,
						blk.meta.ULID.String(),
						[]labels.Labels{blk.extLset},
						onClose,
						blockClient,
						shardMatcher,
						false,
						s.metrics.emptyPostingCount.WithLabelValues(tenant),
						nil,
						nil,
					)
				} else {
					resp = newLazyRespSet(

View on GitHub (pinned to 35b8b99117)