thanos-io/thanos · error

proxy Exemplars

Error message

proxy Exemplars

What it means

Exemplars fans the request out through the proxy to all store APIs and wraps any fan-out error with errors.Wrap(err, "proxy Exemplars"). It indicates the underlying proxy.Exemplars call failed (e.g. InvalidArgument from no selectors, or per-store Aborted errors) before any data could be assembled.

Solutions

1. Read the wrapped cause below 'proxy Exemplars:' in the error string to identify the real failure.
2. Ensure the query has at least one non-external-label matcher (see pkg/exemplars/proxy.go:76).
3. Verify store endpoints support the Exemplars gRPC method and are reachable.
4. If partial data is acceptable, use a partial-response strategy that tolerates store failures instead of ABORT.

data, _, err := api.Exemplars(ctx, req)
if err != nil {
	if st, ok := status.FromError(err); ok && st.Code() == codes.InvalidArgument {
		return fmt.Errorf("bad exemplars query: %w", err)
	}
	return fmt.Errorf("exemplars fanout failed: %w", err)
}

When it happens

Trigger: Calling the ThanOS Query Frontend/Query Exemplars API where rr.proxy.Exemplars returns a non-nil error — e.g. all stores failed under ABORT strategy, or the proxy returned codes.InvalidArgument for empty selectors.

Common situations: Hitting /api/v1/exemplars against stores that don't support exemplars, misconfigured store addresses, or queries whose selectors are entirely stripped as external labels (leading to the proxy's InvalidArgument).

Related errors

• Aborted
• %s
• empty exemplars data
• Aborted
• InvalidArgument

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/529c94e13653267e. Report an issue: GitHub.

Appendix: source

Thrown at pkg/exemplars/exemplars.go:90

	c := &GRPCClient{
		proxy:         es,
		replicaLabels: map[string]struct{}{},
	}

	for _, label := range replicaLabels {
		c.replicaLabels[label] = struct{}{}
	}
	return c
}

func (rr *GRPCClient) Exemplars(ctx context.Context, req *exemplarspb.ExemplarsRequest) ([]*exemplarspb.ExemplarData, annotations.Annotations, error) {
	span, ctx := tracing.StartSpan(ctx, "exemplar_grpc_request")
	defer span.Finish()

	resp := &exemplarsServer{ctx: ctx}

	if err := rr.proxy.Exemplars(req, resp); err != nil {
		return nil, nil, errors.Wrap(err, "proxy Exemplars")
	}

	if resp.data == nil {
		return make([]*exemplarspb.ExemplarData, 0), resp.warnings, nil
	}

	resp.data = dedupExemplarsResponse(resp.data, rr.replicaLabels)
	return resp.data, resp.warnings, nil
}

func dedupExemplarsResponse(exemplarsData []*exemplarspb.ExemplarData, replicaLabels map[string]struct{}) []*exemplarspb.ExemplarData {
	if len(exemplarsData) == 0 {
		return exemplarsData
	}

	// Deduplicate series labels.
	hashToExemplar := make(map[uint64]*exemplarspb.ExemplarData)
	for _, e := range exemplarsData {

View on GitHub (pinned to 35b8b99117)