thanos-io/thanos · error

Aborted

Aborted

Error message

fetching exemplars from exemplars client %v

What it means

exemplarsStream.receive calls the upstream store's Exemplars RPC and wraps any immediate RPC error with "fetching exemplars from exemplars client %v". Under PartialResponseStrategy_ABORT the wrapped error is returned (surfacing as codes.Aborted); otherwise it is forwarded to the client as a warning response and the stream continues.

Solutions

1. Read the wrapped cause and client address in the message; health-check that specific store endpoint.
2. Upgrade stores that return Unimplemented for the Exemplars method.
3. If failures are tolerable, use a non-ABORT partial-response strategy so it degrades to a warning.
4. Remove or repair unreachable stores from the store set.

data, warnings, err := api.Exemplars(ctx, req)
if err != nil {
	// store unreachable: degrade to empty result + alert instead of failing the page
	log.Warnf("exemplars unavailable: %v", err)
	return nil, err
}

When it happens

Trigger: stream.client.Exemplars(ctx, stream.request) fails at call time — store unreachable, DNS failure, TLS error, storeAPI not supporting the Exemplars method (Unimplemented), or context deadline exceeded during fan-out at pkg/exemplars/proxy.go:177.

Common situations: Dead store gateways in the store set, stores running versions without exemplar support, network partition between query and store, per-RPC timeouts too short.

Related errors

• proxy Exemplars
• %s
• empty exemplars data
• Aborted
• InvalidArgument

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/488ca26d9972efd0. Report an issue: GitHub.

Appendix: source

Thrown at pkg/exemplars/proxy.go:177

		}
	}

	return nil
}

func containsLabelName(name string, sets []labels.Labels) bool {
	for _, ls := range sets {
		if ls.Get(name) != "" {
			return true
		}
	}
	return false
}

func (stream *exemplarsStream) receive(ctx context.Context) error {
	exemplars, err := stream.client.Exemplars(ctx, stream.request)
	if err != nil {
		err = errors.Wrapf(err, "fetching exemplars from exemplars client %v", stream.client)

		if stream.request.PartialResponseStrategy == storepb.PartialResponseStrategy_ABORT {
			return err
		}

		if serr := stream.server.Send(exemplarspb.NewWarningExemplarsResponse(err)); serr != nil {
			return serr
		}
		// Not an error if response strategy is warning.
		return nil
	}

	for {
		exemplar, err := exemplars.Recv()
		if err == io.EOF {
			return nil
		}

View on GitHub (pinned to 35b8b99117)