thanos-io/thanos · error

expand

Error message

expand

What it means

After mergeFetchedPostings builds the merged postings set, ExpandPostingsWithContext materializes it into a concrete list of posting IDs; any failure (including context cancellation mid-expansion) is wrapped as "expand". This is Thanos' lazy-postings expansion step that avoids materializing huge posting lists unless needed.

Solutions

1. Check if the wrapped error is context cancellation/deadline exceeded and raise the query timeout.
2. Narrow the label matchers to reduce the size of the postings set.
3. Use time-based partitioning (query smaller time ranges) to shrink matching series.
4. Increase store-gateway resources/limits if expansion routinely fails on large blocks.

// keep posting expansion small before querying
if len(matchers) == 0 || allMatchersTooBroad(matchers) {
  return errors.New("matchers too broad, narrow the selector")
}

if err != nil && strings.Contains(err.Error(), "expand") {
  if errors.Is(errors.Unwrap(err), context.DeadlineExceeded) {
    // increase timeout or narrow query
  }
}

When it happens

Trigger: A Series query whose merged postings are very large, causing ExpandPostingsWithContext to run long enough that the request context is cancelled, or an internal expansion failure.

Common situations: Broad matchers like {job="x"} over huge blocks expanding millions of postings; Prometheus query timeouts killing the gRPC context mid-expansion; store-gateway memory pressure.

Understand the failure class

Background: Request timed out: what client-side request timeouts mean across libraries (Request timed out, TIMED_OUT, APITimeoutError) — this error's family across 39 libraries.

Related errors

• get series
• symbols
• get postings
• reading
• read got bytes instead of 4

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/e8ea113054ce78d5. Report an issue: GitHub.

Appendix: source

Thrown at pkg/store/lazy_postings.go:316

	keys, lazyMatchers := keysToFetchFromPostingGroups(postingGroups)
	fetchedPostings, closeFns, err := r.fetchPostings(ctx, keys, bytesLimiter, tenant)
	defer func() {
		for _, closeFn := range closeFns {
			closeFn()
		}
	}()
	if err != nil {
		return nil, nil, errors.Wrap(err, "get postings")
	}

	result := mergeFetchedPostings(ctx, fetchedPostings, postingGroups)
	if err := ctx.Err(); err != nil {
		return nil, nil, err
	}
	ps, err := ExpandPostingsWithContext(ctx, result)
	r.postings = ps
	if err != nil {
		return nil, nil, errors.Wrap(err, "expand")
	}
	return ps, lazyMatchers, nil
}

func mergeFetchedPostings(ctx context.Context, fetchedPostings []index.Postings, postingGroups []*postingGroup) index.Postings {
	// Get "add" and "remove" postings from groups. We iterate over postingGroups and their keys
	// again, and this is exactly the same order as before (when building the groups), so we can simply
	// use one incrementing index to fetch postings from returned slice.
	postingIndex := 0

	var groupAdds, groupRemovals []index.Postings
	for _, g := range postingGroups {
		if g.lazy {
			continue
		}
		// We cannot add empty set to groupAdds, since they are intersected.
		if len(g.addKeys) > 0 {
			toMerge := make([]index.Postings, 0, len(g.addKeys))

View on GitHub (pinned to 35b8b99117)