thanos-io/thanos · error

proxy Rules

Error message

proxy Rules

What it means

The Rules gRPC handler wraps any error returned by rr.proxy.Rules(req, resp) — the call that fans the rules request out to underlying rule managers/upstream rulers — with the message 'proxy Rules'. The actual cause is in the wrapped error chain; this wrapper only marks the proxy stage of the request.

Solutions

1. Read the wrapped cause in the error chain and fix that specific failure first
2. Verify all upstream ruler endpoints are healthy and reachable
3. Retry the request with a longer context timeout if the upstream was slow

groups, warnings, err := client.Rules(ctx, req)
if err != nil {
    var proxyErr interface{ Unwrap() error }
    if errors.As(err, &proxyErr) {
        log.Printf("rules proxy failed: %v", errors.Unwrap(err))
    }
    return err
}

When it happens

Trigger: Calling the Rules API (client.Rules or the gRPC Rules endpoint) when the internal proxy call fails: upstream ruler unreachable, context cancelled, or a rulesServer.Send error (e.g. 'no group') surfaces from resp.

Common situations: Querying rules while a backend ruler is down; request timeout against a slow upstream; a forwarded response violating the rulespb contract.

Related errors

• proxy Exemplars
• proxy MetricMetadata
• fetching metric metadata from metadata client
• no group
• fetching tsdb statistics from status client

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/cc13b7cfae4105bd. Report an issue: GitHub.

Appendix: source

Thrown at pkg/rules/rules.go:61

	c := &GRPCClient{
		proxy:         rs,
		replicaLabels: map[string]struct{}{},
	}

	for _, label := range replicaLabels {
		c.replicaLabels[label] = struct{}{}
	}
	return c
}

func (rr *GRPCClient) Rules(ctx context.Context, req *rulespb.RulesRequest) (*rulespb.RuleGroups, annotations.Annotations, error) {
	span, ctx := tracing.StartSpan(ctx, "rules_request")
	defer span.Finish()

	resp := &rulesServer{ctx: ctx}

	if err := rr.proxy.Rules(req, resp); err != nil {
		return nil, nil, errors.Wrap(err, "proxy Rules")
	}

	var err error
	matcherSets := make([][]*labels.Matcher, len(req.MatcherString))
	for i, s := range req.MatcherString {
		matcherSets[i], err = extpromql.ParseMetricSelector(s)
		if err != nil {
			return nil, nil, errors.Wrap(err, "parser ParseMetricSelector")
		}
	}

	resp.groups = filterRulesByMatchers(resp.groups, matcherSets)
	resp.groups = filterRulesByNamesAndFile(resp.groups, req.RuleName, req.RuleGroup, req.File)

	// TODO(bwplotka): Move to SortInterface with equal method and heap.
	resp.groups = dedupGroups(resp.groups)
	for _, g := range resp.groups {
		g.Rules = dedupRules(g.Rules, rr.replicaLabels)

View on GitHub (pinned to 35b8b99117)