thanos-io/thanos · error

expanded matching posting

Error message

expanded matching posting

What it means

Wraps a failure from IndexReader.ExpandedPostings inside blockSeriesClient.ExpandPostings. Expanded postings translate label matchers into posting lists via the TSDB index; failures come from the index reader fetching or decoding posting/symbol data from the block's index.

Solutions

1. Inspect the wrapped root error to distinguish network/object-store issues from index corruption.
2. Validate that label matchers do not use an empty label name ('' is rejected as invalid).
3. If corruption is indicated, verify the block with 'thanos tools bucket verify' and remove/restore the bad block.
4. Retry transient object-store failures; check bucket rate limits and store gateway logs.

Example fix

// before
promclient.NewMatcherMatcher(*labels.MustNewMatcher(labels.MatchEqual, "", "value")) // invalid
// after
promclient.NewMatcherMatcher(*labels.MustNewMatcher(labels.MatchEqual, "job", "value"))

// Validate matchers client-side before querying
for _, m := range matchers {
	if m.Name == "" {
		return errors.New("label matcher name must not be empty")
	}
}

func validMatchers(ms []*labels.Matcher) bool {
	for _, m := range ms {
		if m == nil || m.Name == "" {
			return false
		}
	}
	return len(ms) > 0
}

result, err := api.Series(ctx, matchers, start, end)
if err != nil && strings.Contains(err.Error(), "expanded matching posting") {
	// classify: corrupt index vs transient storage error
	if isTransient(err) {
		time.Sleep(backoff)
		result, err = api.Series(ctx, matchers, start, end)
	}
}

When it happens

Trigger: Series() on the store API calls ExpandPostings with label matchers, and the underlying indexr.ExpandedPostings call fails: index fetch from object storage fails, posting data is corrupt, matchers reference an invalid name (empty label name), or the bytes limiter rejects the request.

Common situations: Corrupted index files in the block, object-store throttling/timeouts during heavy queries, clients sending matchers with empty label names, query fan-out hitting blocks still being deleted.

Understand the failure class

Background: Database query failed: Internal Server Error 500s wrapping SQL, Prisma, and connection failures — what to check first — this error's family across 16 libraries.

Related errors

• postings
• postings entries must be in increasing order, current
• input block index not valid
• output block index not valid
• index contains postings with out of order labels

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/2920b64d05951fce. Report an issue: GitHub.

Appendix: source

Thrown at pkg/store/bucket.go:1214

		if matchers[i].Type == matchers[j].Type {
			if matchers[i].Name == matchers[j].Name {
				return matchers[i].Value < matchers[j].Value
			}
			return matchers[i].Name < matchers[j].Name
		}
		return matchers[i].Type < matchers[j].Type
	})

	return matchers
}

func (b *blockSeriesClient) ExpandPostings(
	matchers sortedMatchers,
	seriesLimiter SeriesLimiter,
) error {
	ps, err := b.indexr.ExpandedPostings(b.ctx, matchers, b.bytesLimiter, b.lazyExpandedPostingEnabled, b.seriesMatchRatio, b.postingGroupMaxKeySeriesRatio, b.lazyExpandedPostingSizeBytes, b.lazyExpandedPostingGroupByReason, b.tenant)
	if err != nil {
		return errors.Wrap(err, "expanded matching posting")
	}

	if ps == nil || len(ps.postings) == 0 {
		b.lazyPostings = emptyLazyPostings
		return nil
	}
	b.lazyPostings = ps

	if b.lazyPostings.lazyExpanded() {
		// Assume lazy expansion could cut actual expanded postings length to 50%.
		b.expandedPostings = make([]storage.SeriesRef, 0, len(b.lazyPostings.postings)/2)
		b.lazyExpandedPostingsCount.Inc()
	} else {
		// If seriesLimit is set, it can be applied here to limit the amount of series.
		// Note: This can only be done when postings are not expanded lazily.
		if b.seriesLimit > 0 && len(b.lazyPostings.postings) > b.seriesLimit {
			b.lazyPostings.postings = b.lazyPostings.postings[:b.seriesLimit]
		}

View on GitHub (pinned to 35b8b99117)