What causes "converting prom matchers to storepb matchers" in thanos-io/thanos?

Validate matchers before calling LabelValues; restrict to =, !=, =~, !~

converting prom matchers to storepb matchers

Error message

converting prom matchers to storepb matchers

What it means

LabelValues converts the caller's PromQL label matchers to storepb matchers before proxying the LabelValues RPC. PromMatchersToMatchers failing (unsupported matcher type/regex) is wrapped as 'converting prom matchers to storepb matchers'.

Solutions

Validate matchers before calling LabelValues; restrict to =, !=, =~, !~
Test the regex compiles as RE2 and is representable in storepb
Log the wrapped cause to find the offending matcher
Update Thanos if a matcher type you need fails conversion

Example fix

// before
m := labels.MustNewMatcher(labels.MatchNotRegexp, "job", "(bad regex[")
// after
m := labels.MustNewMatcher(labels.MatchNotRegexp, "job", "other-.*")

Defensive patterns

Strategy: validation

Validate before calling

func validateMatchers(ms []*labels.Matcher) error {
    for _, m := range ms {
        if _, err := regexp.Compile(m.Value); err != nil {
            return fmt.Errorf("matcher %s=%q invalid: %w", m.Name, m.Value, err)
        }
    }
    return nil
}

Type guard

func safeMatcher(m *labels.Matcher) bool {
    t := m.Type
    return t == labels.MatchEqual || t == labels.MatchNotEqual ||
        t == labels.MatchRegexp || t == labels.MatchNotRegexp
}

Try / catch

vals, warns, err := q.LabelValues(ctx, name, hints, matchers...)
if err != nil && strings.Contains(err.Error(), "converting prom matchers") {
    return nil, fmt.Errorf("invalid label matchers: %w", err)
}

Prevention

Sanitize regex input from API consumers
Restrict to standard match types
Test matchers against Prometheus' parser before issuing

When it happens

Trigger: Calling the storage API LabelValues with matchers that storepb cannot encode, e.g. non-standard match types produced by custom client code.

Common situations: Custom integrations constructing labels.Matcher programmatically with invalid regex; API clients posting unusual label match selectors.

Understand the failure class

Background: "Invalid ... format", "must be in format X", "does not look like a ..." — invalid argument format errors across CLI tools and libraries — this error's family across 17 libraries.

Related errors

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/ef25283718d6e899. Report an issue: GitHub.

Appendix: source

Thrown at pkg/query/querier.go:438

		q.maxt,
		aggrs,
		warns,
	)

	return dedup.NewSeriesSet(set, hints.Func, q.deduplicationFunc), resp.seriesSetStats, nil
}

// LabelValues returns all potential values for a label name.
func (q *querier) LabelValues(ctx context.Context, name string, hints *storage.LabelHints, matchers ...*labels.Matcher) ([]string, annotations.Annotations, error) {
	span, ctx := tracing.StartSpan(ctx, "querier_label_values")
	defer span.Finish()

	// TODO(bwplotka): Pass it using the SeriesRequest instead of relying on context.
	ctx = context.WithValue(ctx, store.StoreMatcherKey, q.storeDebugMatchers)

	pbMatchers, err := storepb.PromMatchersToMatchers(matchers...)
	if err != nil {
		return nil, nil, errors.Wrap(err, "converting prom matchers to storepb matchers")
	}

	if hints == nil {
		hints = &storage.LabelHints{}
	}

	req := &storepb.LabelValuesRequest{
		Label:                   name,
		PartialResponseStrategy: q.partialResponseStrategy,
		Start:                   q.mint,
		End:                     q.maxt,
		Matchers:                pbMatchers,
		Limit:                   int64(hints.Limit),
	}

	if q.isDedupEnabled() {
		req.WithoutReplicaLabels = q.replicaLabels
	}

View on GitHub (pinned to 35b8b99117)