{"record":{"id":"9a1fd47fc24bacdd","repo":"thanos-io/thanos","slug":"unknown","errorCode":"Unknown","errorMessage":"send exemplars response","messagePattern":"send exemplars response","errorType":"error_code","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"pkg/exemplars/proxy.go","lineNumber":158,"sourceCode":"\t\t_ = g.Wait()\n\t\tclose(respChan)\n\t}()\n\n\tfor resp := range respChan {\n\t\texemplars = append(exemplars, resp)\n\t}\n\n\tif err := g.Wait(); err != nil {\n\t\tlevel.Error(s.logger).Log(\"err\", err)\n\t\treturn err\n\t}\n\n\tfor _, e := range exemplars {\n\t\ttracing.DoInSpan(srv.Context(), \"send_exemplars_response\", func(_ context.Context) {\n\t\t\terr = srv.Send(exemplarspb.NewExemplarsResponse(e))\n\t\t})\n\t\tif err != nil {\n\t\t\treturn status.Error(codes.Unknown, errors.Wrap(err, \"send exemplars response\").Error())\n\t\t}\n\t}\n\n\treturn nil\n}\n\nfunc containsLabelName(name string, sets []labels.Labels) bool {\n\tfor _, ls := range sets {\n\t\tif ls.Get(name) != \"\" {\n\t\t\treturn true\n\t\t}\n\t}\n\treturn false\n}\n\nfunc (stream *exemplarsStream) receive(ctx context.Context) error {\n\texemplars, err := stream.client.Exemplars(ctx, stream.request)\n\tif err != nil {","sourceCodeStart":140,"sourceCodeEnd":176,"githubUrl":"https://github.com/thanos-io/thanos/blob/35b8b991177def87ed52dcf10f9b6d87f07282c8/pkg/exemplars/proxy.go#L140-L176","documentation":"After collecting exemplars from stores, the proxy streams each back to the client with srv.Send; a failure there is wrapped as \"send exemplars response\" and returned as codes.Unknown. This means the downstream gRPC client connection broke or the RPC was cancelled mid-response.","triggerScenarios":"Client disconnects, RPC deadline/cancel, or transport errors while srv.Send(exemplarspb.NewExemplarsResponse(e)) is called inside the send_exemplars_response span at pkg/exemplars/proxy.go:158.","commonSituations":"HTTP client timeouts on /api/v1/exemplars closing the gRPC stream early, query-frontends with short timeouts on large exemplar responses, network interruptions between query and client.","solutions":["Increase the client's request timeout for exemplars queries on large result sets.","Check client-side cancellation: the user/UI may be aborting the request.","Inspect network stability between the client and the query node (proxy/load-balancer idle timeouts).","Reduce response size with tighter matchers/time ranges so streaming completes faster."],"exampleFix":null,"handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"data, _, err := api.Exemplars(ctx, req)\nif err != nil && strings.Contains(err.Error(), \"send exemplars response\") {\n\t// client/transport dropped mid-stream: back off and retry once\n\ttime.Sleep(time.Second)\n\treturn api.Exemplars(ctx, req)\n}","preventionTips":["Set client timeouts generously for exemplars endpoints.","Avoid proxy/LB idle timeouts shorter than the expected query duration.","Narrow matchers/time range to shrink streamed responses."],"tags":["go","grpc","streaming","exemplars","client-disconnect"],"backgroundTag":"broken-pipe","analyzedSha":"35b8b991177def87ed52dcf10f9b6d87f07282c8","analyzedAt":"2026-09-07T01:49:59.689Z","contentChangedAt":"2026-09-07T01:49:59.689Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}