{"record":{"id":"b35b4591f723f7b4","repo":"thanos-io/thanos","slug":"iterate-series-for-block-s","errorCode":null,"errorMessage":"iterate series for block %s","messagePattern":"iterate series for block (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"pkg/store/bucket.go","lineNumber":1994,"sourceCode":"\n\t\t\t\tif err := blockClient.ExpandPostings(\n\t\t\t\t\tsortedReqSeriesMatchersNoExtLabels,\n\t\t\t\t\tseriesLimiter,\n\t\t\t\t); err != nil {\n\t\t\t\t\treturn err\n\t\t\t\t}\n\n\t\t\t\t// Extract label names from all series. Many label names will be the same, so we need to deduplicate them.\n\t\t\t\t// Note that label names will already include external labels (passed to blockSeries), so we don't need\n\t\t\t\t// to add them again.\n\t\t\t\tlabelNames := map[string]struct{}{}\n\t\t\t\tfor {\n\t\t\t\t\tls, err := blockClient.Recv()\n\t\t\t\t\tif err == io.EOF {\n\t\t\t\t\t\tbreak\n\t\t\t\t\t}\n\t\t\t\t\tif err != nil {\n\t\t\t\t\t\treturn errors.Wrapf(err, \"iterate series for block %s\", b.meta.ULID)\n\t\t\t\t\t}\n\n\t\t\t\t\tif ls.GetWarning() != \"\" {\n\t\t\t\t\t\treturn errors.Wrapf(errors.New(ls.GetWarning()), \"iterate series for block %s\", b.meta.ULID)\n\t\t\t\t\t}\n\t\t\t\t\tif ls.GetSeries() == nil {\n\t\t\t\t\t\tcontinue\n\t\t\t\t\t}\n\t\t\t\t\tfor _, l := range ls.GetSeries().Labels {\n\t\t\t\t\t\tlabelNames[l.Name] = struct{}{}\n\t\t\t\t\t}\n\t\t\t\t}\n\n\t\t\t\tresult = make([]string, 0, len(labelNames))\n\t\t\t\tfor n := range labelNames {\n\t\t\t\t\tresult = append(result, n)\n\t\t\t\t}\n\t\t\t\tsort.Strings(result)","sourceCodeStart":1976,"sourceCodeEnd":2012,"githubUrl":"https://github.com/thanos-io/thanos/blob/35b8b991177def87ed52dcf10f9b6d87f07282c8/pkg/store/bucket.go#L1976-L2012","documentation":"During LabelNames, the store gateway fans out to per-block clients (StoreAPI series streams) and reads responses with blockClient.Recv(). Non-EOF receive errors for a block are wrapped with the block's ULID — the underlying per-block series RPC failed mid-stream.","triggerScenarios":"blockClient.Recv() returns a transport/gRPC error (Unavailable, DeadlineExceeded, Canceled) while iterating series for a block: remote store crashed, network interruption, per-request deadline hit.","commonSituations":"Store gateway to sidecar/remote store network blips during heavy queries; block served by an instance that went away; deadline too short for huge blocks.","solutions":["Retry the LabelNames request; transient Unavailable errors usually clear.","Check logs/health of the store instance serving that block ULID.","Increase the request deadline or limit concurrency if the error is DeadlineExceeded.","Verify block availability; remove or re-sync stale block metadata if the block no longer exists."],"exampleFix":null,"handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"if st, ok := status.FromError(err); ok && st.Code() == codes.Unavailable {\n    time.Sleep(backoff)\n    // retry the LabelNames request\n}","preventionTips":["Set adequate deadlines for large label-name queries","Monitor store-to-store gRPC availability","Keep block metadata synchronized to avoid querying removed blocks"],"tags":["grpc","streaming","thanos-store","network"],"backgroundTag":"api-request-failed","analyzedSha":"35b8b991177def87ed52dcf10f9b6d87f07282c8","analyzedAt":"2026-09-07T01:49:59.689Z","contentChangedAt":"2026-09-07T01:49:59.689Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}