thanos-io/thanos · error

InvalidArgument

InvalidArgument

Error message

no matchers specified (excluding external labels)

What it means

The exemplars proxy rejects requests whose selector list is empty after external labels are stripped, returning codes.InvalidArgument with "no matchers specified (excluding external labels)". Exemplar queries require at least one real matcher; labels injected as external labels don't count.

Solutions

1. Add at least one matcher that is not covered by external labels, e.g. `{job="prometheus"}`.
2. Review the query-frontend external-label configuration so user matchers are not being stripped.
3. Validate selectors client-side before calling the Exemplars API.
4. If the call is intentional fallback behavior, guard the caller (the anonymous wrapper) to skip exemplars queries with no selectors.

Example fix

// before
queryExemplars("{tenant_id=\"x\"}") // fully stripped as external label
// after
queryExemplars("{tenant_id=\"x\", job=\"node\"}")

func hasSelectors(query string, externalLabels labels.Labels) error {
	sel, err := parser.ParseMetricSelector(query)
	if err != nil {
		return err
	}
	for _, m := range sel {
		if externalLabels.Get(m.Name) == "" || externalLabels.Get(m.Name) != m.Value {
			return nil // at least one non-external matcher
		}
	}
	return errors.New("no matchers specified (excluding external labels)")
}

When it happens

Trigger: Calling proxy.Exemplars where ExtractSelectors/label matching yields zero selectors — e.g. a query whose only matchers equal the configured external labels, so they are all removed before validation at pkg/exemplars/proxy.go:76.

Common situations: Queries like `{__name__="up"}` where the whole selector is stripped as external-label overlap, frontend configs with overly broad external-label replication rules, or clients sending empty matcher lists.

Understand the failure class

Background: "missing required argument" and "the following required arguments were not provided": what required-argument errors mean and how to fix them — this error's family across 20 libraries.

Related errors

• %s
• empty exemplars data
• proxy Exemplars
• Aborted
• Unknown

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/a30ec818bd7046f3. Report an issue: GitHub.

Appendix: source

Thrown at pkg/exemplars/proxy.go:76

func (s *Proxy) Exemplars(req *exemplarspb.ExemplarsRequest, srv exemplarspb.Exemplars_ExemplarsServer) error {
	span, ctx := tracing.StartSpan(srv.Context(), "proxy_exemplars")
	defer span.Finish()

	expr, err := extpromql.ParseExpr(req.Query)
	if err != nil {
		return err
	}

	match, selectors := selectorsMatchesExternalLabels(parser.ExtractSelectors(expr), s.selectorLabels)

	// There is no matched selectors for this thanos query.
	if !match {
		return nil
	}

	if len(selectors) == 0 {
		return status.Error(codes.InvalidArgument, errors.New("no matchers specified (excluding external labels)").Error())
	}

	var (
		g, gctx   = errgroup.WithContext(ctx)
		respChan  = make(chan *exemplarspb.ExemplarData, 10)
		exemplars []*exemplarspb.ExemplarData
	)

	queryParts := make([]string, 0)
	labelMatchers := make([]string, 0)
	for _, st := range s.exemplars() {
		queryParts = queryParts[:0]

		for _, matcherSet := range selectors {
			extLbls := st.LabelSets
			if !store.LabelSetsMatch(matcherSet, extLbls...) {
				continue
			}

View on GitHub (pinned to 35b8b99117)