thanos-io/thanos · error

failed to convert matchers

Error message

failed to convert matchers

What it means

While serving TSDB statistics, the sidecar converts protobuf store matchers to Prometheus matchers via storepb.MatchersToPromMatchers. If any matcher has an invalid type or value, conversion fails and the RPC returns 'failed to convert matchers' wrapped with the underlying reason.

Solutions

1. Fix the client to send valid MatchType values (EQ, NEQ, RE, NRE) in each LabelMatcher
2. Validate regex matchers client-side before sending (regexp.Compile)
3. Align protobuf/store API versions between client and sidecar
4. Check the wrapped inner error in logs to identify the exact offending matcher

Example fix

// before
matcher := &storepb.LabelMatcher{Name: "job", Value: "node"}   // MatchType unset
// after
matcher := &storepb.LabelMatcher{Type: storepb.LabelMatcher_EQ, Name: "job", Value: "node"}

for _, m := range matchers {
    if m.Type < storepb.LabelMatcher_EQ || m.Type > storepb.LabelMatcher_NRE {
        return fmt.Errorf("invalid matcher type %v for label %s", m.Type, m.Name)
    }
    if m.Type == storepb.LabelMatcher_RE || m.Type == storepb.LabelMatcher_NRE {
        if _, err := regexp.Compile(m.Value); err != nil {
            return fmt.Errorf("invalid regex %q: %v", m.Value, err)
        }
    }
}

func validMatcher(m *storepb.LabelMatcher) bool {
    switch m.Type {
    case storepb.LabelMatcher_EQ, storepb.LabelMatcher_NEQ,
        storepb.LabelMatcher_RE, storepb.LabelMatcher_NRE:
        if m.Type == storepb.LabelMatcher_RE || m.Type == storepb.LabelMatcher_NRE {
            _, err := regexp.Compile(m.Value)
            return err == nil
        }
        return true
    }
    return false
}

promMatchers, err := storepb.MatchersToPromMatchers(matchers...)
if err != nil {
    return nil, status.Errorf(codes.InvalidArgument, "bad matcher: %v", err)
}

When it happens

Trigger: errors.Wrap inside TSDBStatisticsGetterFunc: storepb.MatchersToPromMatchers(matchers...) errors because a LabelMatcher uses an unknown MatchType or an invalid regex value sent by the client.

Common situations: Custom tooling crafting LabelMatcher protos with a zero-value MatchType; clients built against a newer protobuf enum than the sidecar understands; malformed regex strings passed through UIs.

Understand the failure class

Background: "Must be a positive integer", "Invalid value", "Unsupported": the invalid-argument-value error family, when a library rejects the value you pass — this error's family across 35 libraries.

Related errors

• failed to convert matchers
• failed to convert matchers
• endpoint
• empty exemplars data
• proto: wrong wireType =

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/2740b4866acb50e9. Report an issue: GitHub.

Appendix: source

Thrown at cmd/thanos/sidecar.go:371

			info.WithStatusInfoFunc(),
		)

		statusSrv := status.NewServer(
			component.Sidecar.String(),
			status.WithTSDBStatisticsGetter(
				status.TSDBStatisticsGetterFunc(func(limit int, matchers []storepb.LabelMatcher) (map[string]tsdb.Stats, error) {
					if !httpProbe.IsReady() {
						return nil, errors.New("not ready")
					}

					ctx, cancel := context.WithTimeout(context.Background(), conf.prometheus.getConfigTimeout)
					defer cancel()

					// Check if external labels match the provided matchers.
					extLabels := m.Labels()
					promMatchers, err := storepb.MatchersToPromMatchers(matchers...)
					if err != nil {
						return nil, errors.Wrap(err, "failed to convert matchers")
					}
					for _, matcher := range promMatchers {
						if !matcher.Matches(extLabels.Get(matcher.Name)) {
							// External labels don't match, return empty result.
							return nil, nil
						}
					}

					statsEntry, err := c.TSDBStatusInGRPC(ctx, conf.prometheus.url, limit)
					if err != nil {
						return nil, errors.Wrap(err, "failed to get tsdb status from prometheus")
					}

					return map[string]tsdb.Stats{
						"": statsEntry.ToTSDBStats(limit),
					}, nil
				}),
			),

View on GitHub (pinned to 35b8b99117)