{"record":{"id":"6e48a598b82c30b5","repo":"thanos-io/thanos","slug":"fetch-postings-for-block-s","errorCode":null,"errorMessage":"fetch postings for block %s","messagePattern":"fetch postings for block (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"pkg/store/bucket.go","lineNumber":1679,"sourceCode":"\t\t\t\ts.metrics.lazyExpandedPostingSeriesOverfetchedSizeBytes,\n\t\t\t\ttenant,\n\t\t\t)\n\n\t\t\tdefer blockClient.Close()\n\n\t\t\tg.Go(func() error {\n\t\t\t\tonClose := func() {\n\t\t\t\t\tmtx.Lock()\n\t\t\t\t\tstats = blockClient.MergeStats(stats)\n\t\t\t\t\tmtx.Unlock()\n\t\t\t\t}\n\n\t\t\t\tif err := blockClient.ExpandPostings(\n\t\t\t\t\tsortedBlockMatchers,\n\t\t\t\t\tseriesLimiter,\n\t\t\t\t); err != nil {\n\t\t\t\t\tonClose()\n\t\t\t\t\treturn errors.Wrapf(err, \"fetch postings for block %s\", blk.meta.ULID)\n\t\t\t\t}\n\n\t\t\t\tvar resp respSet\n\t\t\t\tif s.sortingStrategy == sortingStrategyStore {\n\t\t\t\t\tresp = newEagerRespSet(\n\t\t\t\t\t\t10*time.Minute,\n\t\t\t\t\t\tblk.meta.ULID.String(),\n\t\t\t\t\t\t[]labels.Labels{blk.extLset},\n\t\t\t\t\t\tonClose,\n\t\t\t\t\t\tblockClient,\n\t\t\t\t\t\tshardMatcher,\n\t\t\t\t\t\tfalse,\n\t\t\t\t\t\ts.metrics.emptyPostingCount.WithLabelValues(tenant),\n\t\t\t\t\t\tnil,\n\t\t\t\t\t\tnil,\n\t\t\t\t\t)\n\t\t\t\t} else {\n\t\t\t\t\tresp = newLazyRespSet(","sourceCodeStart":1661,"sourceCodeEnd":1697,"githubUrl":"https://github.com/thanos-io/thanos/blob/35b8b991177def87ed52dcf10f9b6d87f07282c8/pkg/store/bucket.go#L1661-L1697","documentation":"BucketStore's Series RPC fetches matching postings (series refs) from each TSDB block via a remote store client (ExpandPostings) and wraps any failure with the block's ULID. It indicates the underlying block client (a StoreAPI gRPC call) failed or was cancelled while resolving label matchers to series for that specific block.","triggerScenarios":"Calling Series on a BucketStore/store gateway when the per-block ExpandPostings gRPC call to a remote store (or the block's index processing) fails: network drop, remote store shutdown, context deadline exceeded during query, or corrupt index preventing postings expansion.","commonSituations":"Queries hitting stale block clients after blocks were deleted/compacted away; store-gateway to store-sidecar network issues; query cancellation/timeouts on large blocks; the seriesLimiter rejecting the request is surfaced through this wrap path too.","solutions":["Retry the query; transient gRPC errors to one block are often resolved by re-querying after store gateway refreshes block metadata.","Check connectivity and logs of the remote store/sidecar serving block blk.meta.ULID (the ULID in the message identifies it).","Verify the block still exists in the bucket and is not pending deletion (compaction); remove stale blocks from the bucket or restart store gateway to refresh its view.","If timeouts recur on large blocks, increase query timeout, enable response compression, or tune posting offsets in-memory cache settings."],"exampleFix":"// before: no per-block resilience, one bad block fails the query\nresp, err := query(ctx, matchers)\n// after: rely on partial-response / retry at query layer\nq, err := querier.New(...).WithSeriesSelector(matchers)... \n// or wrap with retry:\nif errors.Is(err, context.DeadlineExceeded) { err = retryQuery(ctx, matchers) }","handlingStrategy":"retry","validationCode":"// Verify block exists and matches before querying:\nmeta, err := bucket.Get(ctx, userBucket, path.Join(blkULID.String(), \"meta.json\"))\nif err != nil { return err } // block unavailable; skip or resync","typeGuard":null,"tryCatchPattern":"if err != nil {\n    var st status.Status\n    if errors.As(err, &st) && st.Code() == codes.Unavailable {\n        // backoff and retry the query\n    }\n}","preventionTips":["Keep store gateway bucket metadata in sync and restart after deleting blocks","Set generous query deadlines for large blocks","Monitor gRPC connectivity between store gateway and remote stores"],"tags":["grpc","thanos-store","postings","network"],"backgroundTag":"api-request-failed","analyzedSha":"35b8b991177def87ed52dcf10f9b6d87f07282c8","analyzedAt":"2026-09-07T01:49:59.689Z","contentChangedAt":"2026-09-07T01:49:59.689Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}