thanos-io/thanos · error

fetch and expand postings

Error message

fetch and expand postings

What it means

During series expansion, the store fetches and expands posting lists for the computed posting groups via fetchLazyExpandedPostings (possibly deferring some matchers lazily). Errors from those posting fetches/intersections — object-store range reads, bytes-limit rejections, index corruption — are wrapped as 'fetch and expand postings'.

Solutions

1. Read the wrapped cause: throttling => add retries/backoff; bytes limit => shrink query or raise --store.grpc.series-max-bytes-limit.
2. Reduce query scope (shorter time range, fewer matchers) to cut posting volume.
3. Fix object-store reliability (timeouts, rate limits) and increase client concurrency limits.
4. If consistent for one block only, validate the block index and remove/re-upload it.

// bound postings work before fetching
if estimatedPostings > maxBytesPerQuery {
	return fmt.Errorf("query needs %d bytes of postings, over limit %d", estimatedPostings, maxBytesPerQuery)
}

if err != nil {
	if errors.Is(err, contextDeadlineExceeded) || isRateLimited(err) {
		// backoff and retry with narrower time range
	}
	return errors.Wrap(err, "fetch and expand postings")
}

When it happens

Trigger: fetchLazyExpandedPostings(...) returns error: posting range reads from objects fail, requested postings exceed the bytes limiter, lazy-expanded postings fetch fails, or index data is corrupt.

Common situations: Store-gateway hitting --store.grpc.series-read-branch / bytes limits on huge queries, S3 throttling during posting fetches, corrupted index chunks, lazy-expanded-postings feature misconfigured.

Understand the failure class

Background: 'Something went wrong' / 'Request failed (500)' / 'HTTP error! status: 404' — what failed HTTP requests actually mean and how to find the real cause — this error's family across 28 libraries.

Related errors

• get posting offset table from object storage of
• copy posting offsets
• read range
• toPostingGroup
• create meta fetcher

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/cdf53e915f0ea5c0. Report an issue: GitHub.

Appendix: source

Thrown at pkg/store/bucket.go:2715

			continue
		}
		postingGroups[i] = pg
		i++
	}
	postingGroups = postingGroups[:i]

	addAllPostings := allRequested && !hasAdds
	// We only need special All postings if there are no other adds. If there are, we can skip fetching
	// special All postings completely.
	if addAllPostings {
		// add group with label to fetch "special All postings".
		name, value := index.AllPostingsKey()
		postingGroups = append(postingGroups, newPostingGroup(true, name, []string{value}, nil))
	}

	ps, err := fetchLazyExpandedPostings(ctx, postingGroups, r, bytesLimiter, addAllPostings, lazyExpandedPostingEnabled, seriesMatchRatio, postingGroupMaxKeySeriesRatio, lazyExpandedPostingSizeBytes, lazyExpandedPostingGroupsByReason, tenant)
	if err != nil {
		return nil, errors.Wrap(err, "fetch and expand postings")
	}
	// If postings still have matchers to be applied lazily, cache expanded postings after filtering series so skip here.
	if !ps.lazyExpanded() {
		r.storeExpandedPostingsToCache(ms, index.NewListPostings(ps.postings), len(ps.postings), tenant)
	}

	if len(ps.postings) > 0 {
		// As of version two all series entries are 16 byte padded. All references
		// we get have to account for that to get the correct offset.
		version, err := r.IndexVersion()
		if err != nil {
			return nil, errors.Wrap(err, "get index version")
		}
		if version >= 2 {
			for i, id := range ps.postings {
				ps.postings[i] = id * 16
			}
		}

View on GitHub (pinned to 35b8b99117)