thanos-io/thanos · warning

tenant is above active series limit

Error message

tenant is above active series limit

What it means

This is an HTTP 429 (Too Many Requests) response sent by Thanos Receive's OTLP ingest handler when the tenant's number of active series has reached the configured limit enforced by the tenant's Limiter. Receive rejects the whole write request rather than accepting partial data, protecting the store from unbounded memory/TSDB growth. The limit is enforced by the local limiter before the request is decoded.

Solutions

1. Reduce the number of unique series the tenant exports (drop/rename high-cardinality labels via relabeling at the client or receive config).
2. Raise the tenant's active series limit in the tenant limit configuration (tenant_limits or the limiter's ActiveSeriesLimit).
3. Wait for stale/idle series to age out so the tenant's active series count drops below the limit.
4. Handle HTTP 429 client-side by backing off and retrying later (honor Retry-After header semantics).

Example fix

// before: unlimited cardinality labels
counter.WithLabelValues(userID, region, pod, queue, endpoint)
// after: bound label cardinality
counter.WithLabelValues(region, normalizedEndpoint)

resp, err := http.Post(url, contentType, body)
if resp.StatusCode == http.StatusTooManyRequests {
    backoffAndRequeue(body)
}

if resp.StatusCode == 429 { /* exponential backoff, then retry */ }

When it happens

Trigger: POSTing an OTLP/HTTP metrics export to Receive's /api/v1/otlp/v1/metrics endpoint while the tenant is at or above its active-series limit (queried from the query service by the limiter); the limiter lookup ran but reported the tenant is over limit ('under' == false).

Common situations: Cardinality explosion from high-cardinality labels; tenant pushing far more series than provisioned; active-series limit configured too low for the workload; a shard that believes stale series are still active after tenant cardinality dropped.

Related errors

• Admin operations are disabled
• write request too large
• error converting OTLP metrics to Prometheus format
• too many timeseries
• too many samples

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/888ec57902442b46. Report an issue: GitHub.

Appendix: source

Thrown at pkg/receive/handler_otlp.go:57

	tracing.DoInSpan(r.Context(), "receive_write_gate_ismyturn", func(ctx context.Context) {
		err = writeGate.Start(r.Context())
	})

	defer writeGate.Done()
	if err != nil {
		level.Error(tLogger).Log("err", err, "msg", "internal server error")
		http.Error(w, err.Error(), http.StatusInternalServerError)
		return
	}

	under, err := h.Limiter.HeadSeriesLimiter().isUnderLimit(tenant)
	if err != nil {
		level.Error(tLogger).Log("msg", "error while limiting", "err", err.Error())
	}

	// Fail request fully if tenant has exceeded set limit.
	if !under {
		http.Error(w, "tenant is above active series limit", http.StatusTooManyRequests)
		return
	}

	requestLimiter := h.Limiter.RequestLimiter()
	if r.ContentLength >= 0 {
		if !requestLimiter.AllowSizeBytes(tenant, r.ContentLength) {
			http.Error(w, "write request too large", http.StatusRequestEntityTooLarge)
			return
		}
	}

	req, err := remote.DecodeOTLPWriteRequest(r)
	if err != nil {
		level.Error(h.logger).Log("msg", "Error decoding remote write request", "err", err.Error())
		http.Error(w, err.Error(), http.StatusBadRequest)
		return
	}

View on GitHub (pinned to 35b8b99117)