thanos-io/thanos · error · ApiError

error retrieving rules

Error message

error retrieving rules: %v

What it means

Returned by the /api/v1/alerts endpoint when client.Rules() — the RPC that fetches rule groups from the upstream Query node — fails. The error is formatted with Errorf into an ErrorInternal (HTTP 500) response. It indicates the alert data could not be retrieved from the backing rules service, not that the HTTP request was malformed.

Solutions

1. Read the %v-wrapped root error to identify transport vs application failure.
2. Check health/connectivity of the Query nodes serving the Rules API.
3. Retry if the cause was DeadlineExceeded or a transient network error.
4. Verify ruler/store-gateway components feeding rule data are healthy.

try { const r = await fetch('/api/v1/alerts'); const b = await r.json(); if (b.errorType === 'internal') scheduleRetry(b.error); } catch (e) { /* network-level failure */ retry(e); }

When it happens

Trigger: Any GET /api/v1/alerts where the underlying Rules(ctx, req) gRPC call returns an error: unreachable query node, canceled request context, or RPC failure inside tracing span 'retrieve_rules'.

Common situations: Thanos Query node down or restarting, request timeout exceeded, network partition between query-frontend and query, rule Ruler/store RPC erroring while evaluating or serving rules.

Related errors

• retrieving targets
• retrieving exemplars
• no query API server reachable
• failed to get tsdb status from prometheus
• proxy MetricMetadata

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/7b11f878b1a252c0. Report an issue: GitHub.

Appendix: source

Thrown at pkg/api/query/v1.go:1434

		span, ctx := tracing.StartSpan(r.Context(), "receive_http_request")
		defer span.Finish()

		var (
			groups   *rulespb.RuleGroups
			warnings annotations.Annotations
			err      error
		)

		// TODO(bwplotka): Allow exactly the same functionality as query API: passing replica, dedup and partial response as HTTP params as well.
		req := &rulespb.RulesRequest{
			Type:                    rulespb.RulesRequest_ALERT,
			PartialResponseStrategy: ps,
		}
		tracing.DoInSpan(ctx, "retrieve_rules", func(ctx context.Context) {
			groups, warnings, err = client.Rules(ctx, req)
		})
		if err != nil {
			return nil, nil, &api.ApiError{Typ: api.ErrorInternal, Err: errors.Errorf("error retrieving rules: %v", err)}, func() {}
		}

		var resp struct {
			Alerts []*rulespb.AlertInstance `json:"alerts"`
		}
		for _, g := range groups.Groups {
			for _, r := range g.Rules {
				a := r.GetAlert()
				if a == nil {
					continue
				}
				resp.Alerts = append(resp.Alerts, a.Alerts...)
			}
		}
		return resp, warnings.AsErrors(), nil, func() {}
	}
}

View on GitHub (pinned to 35b8b99117)