{"record":{"id":"0d4993a261c27acb","repo":"thanos-io/thanos","slug":"send-series-response","errorCode":null,"errorMessage":"send series response","messagePattern":"send series response","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"pkg/store/proxy.go","lineNumber":406,"sourceCode":"\tif s.enableDedup {\n\t\trespHeap = NewResponseDeduplicator(respHeap)\n\t}\n\n\ti := 0\n\tfor respHeap.Next() {\n\t\ti++\n\t\tif r.Limit > 0 && i > int(r.Limit) {\n\t\t\tbreak\n\t\t}\n\t\tresp := respHeap.At()\n\n\t\tif resp.GetWarning() != \"\" && (r.PartialResponseDisabled || r.PartialResponseStrategy == storepb.PartialResponseStrategy_ABORT) {\n\t\t\treturn status.Error(codes.Aborted, resp.GetWarning())\n\t\t}\n\n\t\tif err := srv.Send(resp); err != nil {\n\t\t\tlevel.Error(reqLogger).Log(\"msg\", \"failed to stream response\", \"error\", err)\n\t\t\treturn status.Error(codes.Unknown, errors.Wrap(err, \"send series response\").Error())\n\t\t}\n\t}\n\n\t// Flush any remaining buffered series from the batchable server.\n\tif f, ok := srv.(flushableServer); ok {\n\t\treturn f.Flush()\n\t}\n\n\treturn nil\n}\n\n// LabelNames returns all known label names.\nfunc (s *ProxyStore) LabelNames(ctx context.Context, originalRequest *storepb.LabelNamesRequest) (*storepb.LabelNamesResponse, error) {\n\t// TODO(bwplotka): This should be part of request logger, otherwise it does not make much sense. Also, could be\n\t// triggered by tracing span to reduce cognitive load.\n\treqLogger := log.With(s.logger, \"component\", \"proxy\")\n\tif s.debugLogging {\n\t\treqLogger = log.With(reqLogger, \"request\", originalRequest.String())","sourceCodeStart":388,"sourceCodeEnd":424,"githubUrl":"https://github.com/thanos-io/thanos/blob/35b8b991177def87ed52dcf10f9b6d87f07282c8/pkg/store/proxy.go#L388-L424","documentation":"Inside the proxy Store's Series handler, each response received from a downstream store is streamed back to the caller via srv.Send. If Send fails (client disconnected, gRPC stream broken, deadline exceeded), the handler logs and returns an Unknown gRPC status wrapping the error as \"send series response\". It indicates the stream to the client broke mid-response, not a downstream store problem.","triggerScenarios":"Client cancels the query or its context expires while responses are being streamed; network interruption between the client and the Query proxy; client-side deadline shorter than the time needed to stream all series.","commonSituations":"Grafana/UI users canceling large queries; grpc client max response size or deadline too small; load balancer idle timeouts killing long streams; query aborted in ABORT mode causing client to close early.","solutions":["Check whether the client canceled — if so this error is benign and can be logged at debug level","Increase client gRPC deadline/timeout to cover full result streaming","Narrow the query (smaller time range, more matchers) to reduce streamed volume","Check network path (LB idle timeout, grpc keepalive settings) for prematurely closed streams"],"exampleFix":"// before: short client deadline kills the stream\nctx, cancel := context.WithTimeout(ctx, 5*time.Second)\n// after: deadline proportional to expected result size\nctx, cancel := context.WithTimeout(ctx, 120*time.Second)","handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"_, err := queryClient.Series(ctx, req)\nif err != nil && strings.Contains(err.Error(), \"send series response\") {\n    // client stream broke: check if ctx was canceled (benign) or network issue (retry with longer deadline)\n    if ctx.Err() != nil { log.Debug(\"query canceled by client\") } else { retry with increased deadline }\n}","preventionTips":["Set client gRPC deadlines longer than expected full-response streaming time","Configure gRPC keepalive and LB idle timeouts above query duration","Reduce result volume with tighter matchers/time ranges","Treat cancellation-induced sends as benign and log at debug level"],"tags":["thanos","grpc","streaming","network"],"backgroundTag":"broken-pipe","analyzedSha":"35b8b991177def87ed52dcf10f9b6d87f07282c8","analyzedAt":"2026-09-07T01:49:59.689Z","contentChangedAt":"2026-09-07T01:49:59.689Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}