{"record":{"id":"2920b64d05951fce","repo":"thanos-io/thanos","slug":"expanded-matching-posting","errorCode":null,"errorMessage":"expanded matching posting","messagePattern":"expanded matching posting","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"pkg/store/bucket.go","lineNumber":1214,"sourceCode":"\t\tif matchers[i].Type == matchers[j].Type {\n\t\t\tif matchers[i].Name == matchers[j].Name {\n\t\t\t\treturn matchers[i].Value < matchers[j].Value\n\t\t\t}\n\t\t\treturn matchers[i].Name < matchers[j].Name\n\t\t}\n\t\treturn matchers[i].Type < matchers[j].Type\n\t})\n\n\treturn matchers\n}\n\nfunc (b *blockSeriesClient) ExpandPostings(\n\tmatchers sortedMatchers,\n\tseriesLimiter SeriesLimiter,\n) error {\n\tps, err := b.indexr.ExpandedPostings(b.ctx, matchers, b.bytesLimiter, b.lazyExpandedPostingEnabled, b.seriesMatchRatio, b.postingGroupMaxKeySeriesRatio, b.lazyExpandedPostingSizeBytes, b.lazyExpandedPostingGroupByReason, b.tenant)\n\tif err != nil {\n\t\treturn errors.Wrap(err, \"expanded matching posting\")\n\t}\n\n\tif ps == nil || len(ps.postings) == 0 {\n\t\tb.lazyPostings = emptyLazyPostings\n\t\treturn nil\n\t}\n\tb.lazyPostings = ps\n\n\tif b.lazyPostings.lazyExpanded() {\n\t\t// Assume lazy expansion could cut actual expanded postings length to 50%.\n\t\tb.expandedPostings = make([]storage.SeriesRef, 0, len(b.lazyPostings.postings)/2)\n\t\tb.lazyExpandedPostingsCount.Inc()\n\t} else {\n\t\t// If seriesLimit is set, it can be applied here to limit the amount of series.\n\t\t// Note: This can only be done when postings are not expanded lazily.\n\t\tif b.seriesLimit > 0 && len(b.lazyPostings.postings) > b.seriesLimit {\n\t\t\tb.lazyPostings.postings = b.lazyPostings.postings[:b.seriesLimit]\n\t\t}","sourceCodeStart":1196,"sourceCodeEnd":1232,"githubUrl":"https://github.com/thanos-io/thanos/blob/35b8b991177def87ed52dcf10f9b6d87f07282c8/pkg/store/bucket.go#L1196-L1232","documentation":"Wraps a failure from IndexReader.ExpandedPostings inside blockSeriesClient.ExpandPostings. Expanded postings translate label matchers into posting lists via the TSDB index; failures come from the index reader fetching or decoding posting/symbol data from the block's index.","triggerScenarios":"Series() on the store API calls ExpandPostings with label matchers, and the underlying indexr.ExpandedPostings call fails: index fetch from object storage fails, posting data is corrupt, matchers reference an invalid name (empty label name), or the bytes limiter rejects the request.","commonSituations":"Corrupted index files in the block, object-store throttling/timeouts during heavy queries, clients sending matchers with empty label names, query fan-out hitting blocks still being deleted.","solutions":["Inspect the wrapped root error to distinguish network/object-store issues from index corruption.","Validate that label matchers do not use an empty label name ('' is rejected as invalid).","If corruption is indicated, verify the block with 'thanos tools bucket verify' and remove/restore the bad block.","Retry transient object-store failures; check bucket rate limits and store gateway logs."],"exampleFix":"// before\npromclient.NewMatcherMatcher(*labels.MustNewMatcher(labels.MatchEqual, \"\", \"value\")) // invalid\n// after\npromclient.NewMatcherMatcher(*labels.MustNewMatcher(labels.MatchEqual, \"job\", \"value\"))","handlingStrategy":"retry","validationCode":"// Validate matchers client-side before querying\nfor _, m := range matchers {\n\tif m.Name == \"\" {\n\t\treturn errors.New(\"label matcher name must not be empty\")\n\t}\n}","typeGuard":"func validMatchers(ms []*labels.Matcher) bool {\n\tfor _, m := range ms {\n\t\tif m == nil || m.Name == \"\" {\n\t\t\treturn false\n\t\t}\n\t}\n\treturn len(ms) > 0\n}","tryCatchPattern":"result, err := api.Series(ctx, matchers, start, end)\nif err != nil && strings.Contains(err.Error(), \"expanded matching posting\") {\n\t// classify: corrupt index vs transient storage error\n\tif isTransient(err) {\n\t\ttime.Sleep(backoff)\n\t\tresult, err = api.Series(ctx, matchers, start, end)\n\t}\n}","preventionTips":["Reject empty label names in query preprocessing.","Set sane query time ranges and matchers to limit posting expansion size.","Run 'thanos tools bucket verify' to catch index corruption early.","Monitor object-store throttling (429s) from store gateways."],"tags":["index","thanos-query","postings"],"backgroundTag":"database-query-failed","analyzedSha":"35b8b991177def87ed52dcf10f9b6d87f07282c8","analyzedAt":"2026-09-07T01:49:59.689Z","contentChangedAt":"2026-09-07T01:49:59.689Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}