{"record":{"id":"cdf53e915f0ea5c0","repo":"thanos-io/thanos","slug":"fetch-and-expand-postings","errorCode":null,"errorMessage":"fetch and expand postings","messagePattern":"fetch and expand postings","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"pkg/store/bucket.go","lineNumber":2715,"sourceCode":"\t\t\tcontinue\n\t\t}\n\t\tpostingGroups[i] = pg\n\t\ti++\n\t}\n\tpostingGroups = postingGroups[:i]\n\n\taddAllPostings := allRequested && !hasAdds\n\t// We only need special All postings if there are no other adds. If there are, we can skip fetching\n\t// special All postings completely.\n\tif addAllPostings {\n\t\t// add group with label to fetch \"special All postings\".\n\t\tname, value := index.AllPostingsKey()\n\t\tpostingGroups = append(postingGroups, newPostingGroup(true, name, []string{value}, nil))\n\t}\n\n\tps, err := fetchLazyExpandedPostings(ctx, postingGroups, r, bytesLimiter, addAllPostings, lazyExpandedPostingEnabled, seriesMatchRatio, postingGroupMaxKeySeriesRatio, lazyExpandedPostingSizeBytes, lazyExpandedPostingGroupsByReason, tenant)\n\tif err != nil {\n\t\treturn nil, errors.Wrap(err, \"fetch and expand postings\")\n\t}\n\t// If postings still have matchers to be applied lazily, cache expanded postings after filtering series so skip here.\n\tif !ps.lazyExpanded() {\n\t\tr.storeExpandedPostingsToCache(ms, index.NewListPostings(ps.postings), len(ps.postings), tenant)\n\t}\n\n\tif len(ps.postings) > 0 {\n\t\t// As of version two all series entries are 16 byte padded. All references\n\t\t// we get have to account for that to get the correct offset.\n\t\tversion, err := r.IndexVersion()\n\t\tif err != nil {\n\t\t\treturn nil, errors.Wrap(err, \"get index version\")\n\t\t}\n\t\tif version >= 2 {\n\t\t\tfor i, id := range ps.postings {\n\t\t\t\tps.postings[i] = id * 16\n\t\t\t}\n\t\t}","sourceCodeStart":2697,"sourceCodeEnd":2733,"githubUrl":"https://github.com/thanos-io/thanos/blob/35b8b991177def87ed52dcf10f9b6d87f07282c8/pkg/store/bucket.go#L2697-L2733","documentation":"During series expansion, the store fetches and expands posting lists for the computed posting groups via fetchLazyExpandedPostings (possibly deferring some matchers lazily). Errors from those posting fetches/intersections — object-store range reads, bytes-limit rejections, index corruption — are wrapped as 'fetch and expand postings'.","triggerScenarios":"fetchLazyExpandedPostings(...) returns error: posting range reads from objects fail, requested postings exceed the bytes limiter, lazy-expanded postings fetch fails, or index data is corrupt.","commonSituations":"Store-gateway hitting --store.grpc.series-read-branch / bytes limits on huge queries, S3 throttling during posting fetches, corrupted index chunks, lazy-expanded-postings feature misconfigured.","solutions":["Read the wrapped cause: throttling => add retries/backoff; bytes limit => shrink query or raise --store.grpc.series-max-bytes-limit.","Reduce query scope (shorter time range, fewer matchers) to cut posting volume.","Fix object-store reliability (timeouts, rate limits) and increase client concurrency limits.","If consistent for one block only, validate the block index and remove/re-upload it."],"exampleFix":null,"handlingStrategy":"retry","validationCode":"// bound postings work before fetching\nif estimatedPostings > maxBytesPerQuery {\n\treturn fmt.Errorf(\"query needs %d bytes of postings, over limit %d\", estimatedPostings, maxBytesPerQuery)\n}","typeGuard":null,"tryCatchPattern":"if err != nil {\n\tif errors.Is(err, contextDeadlineExceeded) || isRateLimited(err) {\n\t\t// backoff and retry with narrower time range\n\t}\n\treturn errors.Wrap(err, \"fetch and expand postings\")\n}","preventionTips":["Set realistic --store.grpc.series-max-bytes-limit values","Shard huge queries across smaller time ranges","Ensure object-store rate limits accommodate posting fetch concurrency"],"tags":["postings","thanos-store-gateway","object-storage"],"backgroundTag":"http-request-failed","analyzedSha":"35b8b991177def87ed52dcf10f9b6d87f07282c8","analyzedAt":"2026-09-07T01:49:59.689Z","contentChangedAt":"2026-09-07T01:49:59.689Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}