thanos-io/thanos · error

failed to convert matchers

Error message

failed to convert matchers

What it means

The status server converts incoming protobuf storepb.LabelMatchers to Prometheus matchers via storepb.MatchersToPromMatchers before applying external-label filtering. If a matcher's type or value cannot be represented as a Prometheus matcher, the error is wrapped as "failed to convert matchers" and the TSDBStats RPC fails.

Solutions

1. Fix the matcher: ensure regex values are valid RE2 (test with a regex validator).
2. Use only supported matcher types (EQ, NEQ, RE, NRE) in the request.
3. Update the client library to match the server's storepb API version.
4. Simplify to exact label matchers ("=" instead of "=~") when regex is unnecessary.

Example fix

// before
matcher: {type: RE, name: "rule", value: "[a-"}   // invalid regex
// after
matcher: {type: RE, name: "rule", value: "[a-z]+"}

// validate matchers client-side before the RPC
re, err := regexp.Compile(m.Value)
if err != nil { return fmt.Errorf("invalid matcher regex %q: %w", m.Value, err) }

func validMatcher(m storepb.LabelMatcher) bool {
  switch m.Type {
  case storepb.LabelMatcher_EQ, storepb.LabelMatcher_NEQ:
    return true
  case storepb.LabelMatcher_RE, storepb.LabelMatcher_NRE:
    _, err := regexp.Compile(m.Value)
    return err == nil
  }
  return false
}

When it happens

Trigger: Calling the TSDB statistics API with a malformed label matcher: unknown matcher type in the protobuf, invalid regex value (uncompilable RE2), or an otherwise invalid matcher value in the request.

Common situations: Hand-crafted gRPC/API requests with wrong matcher type enums; a client sending a regex matcher with syntactically invalid regex like "[a-"; version mismatch between client storepb API and server.

Related errors

• failed to convert matchers
• failed to convert matchers
• endpoint
• validate relabel config
• InvalidArgument

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/798884a382da7fca. Report an issue: GitHub.

Appendix: source

Thrown at cmd/thanos/rule.go:800

			info.WithStatusInfoFunc(),
		)
		storeServer := store.NewLimitedStoreServer(store.NewInstrumentedStoreServer(reg, tsdbStore), reg, conf.storeRateLimits)
		options = append(options, grpcserver.WithServer(store.RegisterStoreServer(storeServer, logger)))

		// Add Status server for TSDB statistics
		statusSrv := status.NewServer(
			component.Rule.String(),
			status.WithTSDBStatisticsGetter(
				status.TSDBStatisticsGetterFunc(func(limit int, matchers []storepb.LabelMatcher) (map[string]tsdb.Stats, error) {
					if !httpProbe.IsReady() {
						return nil, errors.New("not ready")
					}

					// Check if external labels match the provided matchers.
					extLabels := conf.lset
					promMatchers, err := storepb.MatchersToPromMatchers(matchers...)
					if err != nil {
						return nil, errors.Wrap(err, "failed to convert matchers")
					}
					for _, matcher := range promMatchers {
						if !matcher.Matches(extLabels.Get(matcher.Name)) {
							// External labels don't match, return empty result.
							return nil, nil
						}
					}

					stats := tsdbDB.Head().Stats(labels.MetricName, limit)
					return map[string]tsdb.Stats{
						"": *stats,
					}, nil
				}),
			),
		)
		options = append(options, grpcserver.WithServer(status.RegisterStatusServer(statusSrv)))
	}

View on GitHub (pinned to 35b8b99117)