thanos-io/thanos · warning

Unknown

Unknown

Error message

send exemplars response

What it means

After collecting exemplars from stores, the proxy streams each back to the client with srv.Send; a failure there is wrapped as "send exemplars response" and returned as codes.Unknown. This means the downstream gRPC client connection broke or the RPC was cancelled mid-response.

Solutions

1. Increase the client's request timeout for exemplars queries on large result sets.
2. Check client-side cancellation: the user/UI may be aborting the request.
3. Inspect network stability between the client and the query node (proxy/load-balancer idle timeouts).
4. Reduce response size with tighter matchers/time ranges so streaming completes faster.

data, _, err := api.Exemplars(ctx, req)
if err != nil && strings.Contains(err.Error(), "send exemplars response") {
	// client/transport dropped mid-stream: back off and retry once
	time.Sleep(time.Second)
	return api.Exemplars(ctx, req)
}

When it happens

Trigger: Client disconnects, RPC deadline/cancel, or transport errors while srv.Send(exemplarspb.NewExemplarsResponse(e)) is called inside the send_exemplars_response span at pkg/exemplars/proxy.go:158.

Common situations: HTTP client timeouts on /api/v1/exemplars closing the gRPC stream early, query-frontends with short timeouts on large exemplar responses, network interruptions between query and client.

Related errors

• %s
• empty exemplars data
• proxy Exemplars
• Aborted
• InvalidArgument

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/9a1fd47fc24bacdd. Report an issue: GitHub.

Appendix: source

Thrown at pkg/exemplars/proxy.go:158

		_ = g.Wait()
		close(respChan)
	}()

	for resp := range respChan {
		exemplars = append(exemplars, resp)
	}

	if err := g.Wait(); err != nil {
		level.Error(s.logger).Log("err", err)
		return err
	}

	for _, e := range exemplars {
		tracing.DoInSpan(srv.Context(), "send_exemplars_response", func(_ context.Context) {
			err = srv.Send(exemplarspb.NewExemplarsResponse(e))
		})
		if err != nil {
			return status.Error(codes.Unknown, errors.Wrap(err, "send exemplars response").Error())
		}
	}

	return nil
}

func containsLabelName(name string, sets []labels.Labels) bool {
	for _, ls := range sets {
		if ls.Get(name) != "" {
			return true
		}
	}
	return false
}

func (stream *exemplarsStream) receive(ctx context.Context) error {
	exemplars, err := stream.client.Exemplars(ctx, stream.request)
	if err != nil {

View on GitHub (pinned to 35b8b99117)