thanos-io/thanos · error

proxy LabelValues()

Error message

proxy LabelValues()

What it means

LabelValues fans out to stores through the gRPC proxy; any error from proxy.LabelValues (transport failure, remote error, deadline) is wrapped as 'proxy LabelValues()'. It's the umbrella error for store-side failures during label value lookup.

Solutions

1. Check store endpoint health/connectivity (grpcurl, logs)
2. Review the wrapped underlying error for which store failed
3. Raise timeouts or reduce label matcher scope for large cardinality labels
4. Ensure network policies/firewalls allow gRPC on store ports

// verify at least one healthy store before LabelValues fan-out
if len(healthyStores(stores)) == 0 {
    return errors.New("no healthy stores for LabelValues")
}

vals, warns, err := q.LabelValues(ctx, name, hints, matchers...)
if err != nil {
    if strings.Contains(err.Error(), "proxy LabelValues()") {
        return retryBackoff(ctx, 3, func() error { return fetchValues() })
    }
    return err
}

When it happens

Trigger: Calling LabelValues while one or more queried stores fail the LabelValues RPC: unreachable endpoints, TLS errors, context deadline exceeded.

Common situations: Store gateway outage, Kubernetes service DNS issues, query frontend timeout too short, network policies blocking gRPC traffic.

Related errors

• proxy LabelNames()
• proxy Series()
• send series response
• building gRPC client
• endpoint

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/69be1c691dd28ab1. Report an issue: GitHub.

Appendix: source

Thrown at pkg/query/querier.go:460

		hints = &storage.LabelHints{}
	}

	req := &storepb.LabelValuesRequest{
		Label:                   name,
		PartialResponseStrategy: q.partialResponseStrategy,
		Start:                   q.mint,
		End:                     q.maxt,
		Matchers:                pbMatchers,
		Limit:                   int64(hints.Limit),
	}

	if q.isDedupEnabled() {
		req.WithoutReplicaLabels = q.replicaLabels
	}

	resp, err := q.proxy.LabelValues(ctx, req)
	if err != nil {
		return nil, nil, errors.Wrap(err, "proxy LabelValues()")
	}

	var warns annotations.Annotations
	for _, w := range resp.Warnings {
		warns.Add(errors.New(w))
	}

	return resp.Values, warns, nil
}

// LabelNames returns all the unique label names present in the block in sorted order constrained
// by the given matchers.
func (q *querier) LabelNames(ctx context.Context, hints *storage.LabelHints, matchers ...*labels.Matcher) ([]string, annotations.Annotations, error) {
	span, ctx := tracing.StartSpan(ctx, "querier_label_names")
	defer span.Finish()

	// TODO(bwplotka): Pass it using the SeriesRequest instead of relying on context.
	ctx = context.WithValue(ctx, store.StoreMatcherKey, q.storeDebugMatchers)

View on GitHub (pinned to 35b8b99117)