{"record":{"id":"4d16561d58f5c64a","repo":"thanos-io/thanos","slug":"fetching-tsdb-statistics-from-status-client-v","errorCode":null,"errorMessage":"fetching tsdb statistics from status client %v","messagePattern":"fetching tsdb statistics from status client (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"pkg/status/proxy.go","lineNumber":102,"sourceCode":"type statsStream struct {\n\tclient  statuspb.StatusClient\n\trequest *statuspb.TSDBStatisticsRequest\n\tchannel chan<- *statuspb.TSDBStatistics\n\tserver  statuspb.Status_TSDBStatisticsServer\n}\n\nfunc (stream *statsStream) receive(ctx context.Context) error {\n\tvar (\n\t\terr         error\n\t\tstatsClient statuspb.Status_TSDBStatisticsClient\n\t)\n\n\ttracing.DoInSpan(ctx, \"receive_tsdb_statistics_stream_request\", func(ctx context.Context) {\n\t\tstatsClient, err = stream.client.TSDBStatistics(ctx, stream.request)\n\t})\n\n\tif err != nil {\n\t\terr = errors.Wrapf(err, \"fetching tsdb statistics from status client %v\", stream.client)\n\n\t\tif stream.request.PartialResponseStrategy == storepb.PartialResponseStrategy_ABORT {\n\t\t\treturn err\n\t\t}\n\n\t\tif serr := stream.server.Send(statuspb.NewWarningTSDBStatisticsResponse(err)); serr != nil {\n\t\t\treturn serr\n\t\t}\n\n\t\t// Not an error if response strategy is warning.\n\t\treturn nil\n\t}\n\n\tfor {\n\t\tresp, err := statsClient.Recv()\n\t\tif err == io.EOF {\n\t\t\treturn nil\n\t\t}","sourceCodeStart":84,"sourceCodeEnd":120,"githubUrl":"https://github.com/thanos-io/thanos/blob/35b8b991177def87ed52dcf10f9b6d87f07282c8/pkg/status/proxy.go#L84-L120","documentation":"The status proxy's receive goroutine opens a TSDBStatistics stream to a store client; if that RPC fails to start, the error is wrapped with the client's address. Under PartialResponseStrategy_ABORT it is a fatal error; otherwise it is downgraded to a warning response sent to the caller.","triggerScenarios":"Streaming TSDB statistics from a store whose gRPC endpoint is unreachable, the store returns an error, or the per-request timeout fires when `stream.client.TSDBStatistics(ctx, stream.request)` is called.","commonSituations":"A store node is down or restarted; wrong/DNS-stale store address; TLS misconfiguration between querier and store; store overloaded and rejecting new RPCs; ABORT strategy makes one bad store fail the whole request.","solutions":["Check connectivity to the store address shown in the error (the wrapped %v).","Restart or replace the unhealthy store node, or remove it from the store set.","Switch PartialResponseStrategy to WARNING if partial results are acceptable.","Inspect store-side logs for the underlying gRPC error (unavailable, deadline, auth)."],"exampleFix":"// before\nreq.PartialResponseStrategy = storepb.PartialResponseStrategy_ABORT\n// after\nreq.PartialResponseStrategy = storepb.PartialResponseStrategy_WARNINGS // tolerate down stores with partial results","handlingStrategy":"fallback","validationCode":null,"typeGuard":null,"tryCatchPattern":"if err != nil {\n    if strings.Contains(err.Error(), \"fetching tsdb statistics from status client\") {\n        logger.Warn(\"store unavailable, continuing with partial stats\", \"store\", storeAddr)\n        return partialStats, nil\n    }\n    return err\n}","preventionTips":["Health-check stores before issuing statistics requests.","Prefer WARNINGS partial response strategy for dashboards.","Alert on store node availability.","Verify DNS/TLS between querier and stores."],"tags":["grpc","proxy","store","network"],"backgroundTag":"upstream-api-error","analyzedSha":"35b8b991177def87ed52dcf10f9b6d87f07282c8","analyzedAt":"2026-09-07T01:49:59.689Z","contentChangedAt":"2026-09-07T01:49:59.689Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}