{"record":{"id":"75f142db0dd769ff","repo":"thanos-io/thanos","slug":"proxy-series","errorCode":null,"errorMessage":"proxy Series()","messagePattern":"proxy Series\\(\\)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"pkg/query/querier.go","lineNumber":400,"sourceCode":"\t\tShardInfo:               q.shardInfo,\n\t\tPartialResponseStrategy: q.partialResponseStrategy,\n\t\tSkipChunks:              q.skipChunks,\n\t\tResponseBatchSize:       int64(q.seriesResponseBatchSize),\n\t\tQueryHints: &storepb.QueryHints{\n\t\t\tProjectionLabels:  hints.ProjectionLabels,\n\t\t\tProjectionInclude: hints.ProjectionInclude,\n\t\t\t// NOTE(GiedriusS): this is what the Thanos parquet gateway uses right now so\n\t\t\t// I just copied/pasted it.\n\t\t\tSeriesHashLabelName: SeriesHashLabelName,\n\t\t},\n\t}\n\tif q.isDedupEnabled() {\n\t\t// Soft ask to sort without replica labels and push them at the end of labelset.\n\t\treq.WithoutReplicaLabels = q.replicaLabels\n\t}\n\n\tif err := q.proxy.Series(&req, resp); err != nil {\n\t\treturn nil, storepb.SeriesStatsCounter{}, errors.Wrap(err, \"proxy Series()\")\n\t}\n\twarns := annotations.New().Merge(resp.warnings)\n\n\tif !q.isDedupEnabled() {\n\t\treturn NewPromSeriesSet(\n\t\t\tnewStoreSeriesSet(resp.seriesSet),\n\t\t\tq.mint,\n\t\t\tq.maxt,\n\t\t\taggrs,\n\t\t\twarns,\n\t\t), resp.seriesSetStats, nil\n\t}\n\n\t// TODO(bwplotka): Move to deduplication on chunk level inside promSeriesSet, similar to what we have in dedup.NewDedupChunkMerger().\n\t// This however require big refactor, caring about correct AggrChunk to iterator conversion and counter reset apply.\n\t// For now we apply simple logic that splits potential overlapping chunks into separate replica series, so we can split the work.\n\tset := NewPromSeriesSet(\n\t\tdedup.NewOverlapSplit(newStoreSeriesSet(resp.seriesSet)),","sourceCodeStart":382,"sourceCodeEnd":418,"githubUrl":"https://github.com/thanos-io/thanos/blob/35b8b991177def87ed52dcf10f9b6d87f07282c8/pkg/query/querier.go#L382-L418","documentation":"selectFn issues a gRPC Series() call through the store proxy (fan-out to store/querier nodes). Any transport, RPC, or remote-store error returned by proxy.Series is wrapped as 'proxy Series()'. This is the umbrella error for store-gRPC failures during a series query.","triggerScenarios":"Querier Select fans out to stores and one or more gRPC Series calls fail: connection refused, TLS issues, remote side returns an error, deadline exceeded.","commonSituations":"Store gateway down or unreachable, DNS/service discovery failures in Kubernetes, TLS certificate mismatch, per-store timeouts on large queries, network partition.","solutions":["Check connectivity to the store endpoints (DNS, ports, TLS)","Inspect store-gateway/sidecar logs for the underlying RPC error","Increase --store.grpc.series-timeout / query timeout if queries are large","Verify store nodes are registered and healthy in the query's store API set"],"exampleFix":"// before: proxy.Series fails on dead store\n// after: check/restart store and confirm with grpc probe\nthanos store endpoints:\n  - store-gateway.thanos:10901\n# verify: grpcurl -plaintext store-gateway.thanos:10901 list","handlingStrategy":"retry","validationCode":"// probe stores before issuing queries\nfor _, s := range stores {\n    if !grpcHealthy(ctx, s.Addr, 2*time.Second) {\n        return fmt.Errorf(\"store %s unhealthy\", s.Addr)\n    }\n}","typeGuard":null,"tryCatchPattern":"if err := ss.Err(); err != nil {\n    if strings.Contains(err.Error(), \"proxy Series()\") {\n        // partial store failure: retry with backoff, or filter dead stores\n        return retryBackoff(ctx, func() error { return runQuery() })\n    }\n    return err\n}","preventionTips":["Health-check store endpoints before fan-out","Set sane gRPC deadlines and series timeouts","Monitor store-gateway/sidecar availability with alerts"],"tags":["thanos","query","grpc","network"],"backgroundTag":"upstream-api-error","analyzedSha":"35b8b991177def87ed52dcf10f9b6d87f07282c8","analyzedAt":"2026-09-07T01:49:59.689Z","contentChangedAt":"2026-09-07T01:49:59.689Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}