{"record":{"id":"888ec57902442b46","repo":"thanos-io/thanos","slug":"tenant-is-above-active-series-limit-888ec5","errorCode":null,"errorMessage":"tenant is above active series limit","messagePattern":"tenant is above active series limit","errorType":"http","errorClass":null,"httpStatus":429,"severity":"warning","filePath":"pkg/receive/handler_otlp.go","lineNumber":57,"sourceCode":"\ttracing.DoInSpan(r.Context(), \"receive_write_gate_ismyturn\", func(ctx context.Context) {\n\t\terr = writeGate.Start(r.Context())\n\t})\n\n\tdefer writeGate.Done()\n\tif err != nil {\n\t\tlevel.Error(tLogger).Log(\"err\", err, \"msg\", \"internal server error\")\n\t\thttp.Error(w, err.Error(), http.StatusInternalServerError)\n\t\treturn\n\t}\n\n\tunder, err := h.Limiter.HeadSeriesLimiter().isUnderLimit(tenant)\n\tif err != nil {\n\t\tlevel.Error(tLogger).Log(\"msg\", \"error while limiting\", \"err\", err.Error())\n\t}\n\n\t// Fail request fully if tenant has exceeded set limit.\n\tif !under {\n\t\thttp.Error(w, \"tenant is above active series limit\", http.StatusTooManyRequests)\n\t\treturn\n\t}\n\n\trequestLimiter := h.Limiter.RequestLimiter()\n\tif r.ContentLength >= 0 {\n\t\tif !requestLimiter.AllowSizeBytes(tenant, r.ContentLength) {\n\t\t\thttp.Error(w, \"write request too large\", http.StatusRequestEntityTooLarge)\n\t\t\treturn\n\t\t}\n\t}\n\n\treq, err := remote.DecodeOTLPWriteRequest(r)\n\tif err != nil {\n\t\tlevel.Error(h.logger).Log(\"msg\", \"Error decoding remote write request\", \"err\", err.Error())\n\t\thttp.Error(w, err.Error(), http.StatusBadRequest)\n\t\treturn\n\t}\n","sourceCodeStart":39,"sourceCodeEnd":75,"githubUrl":"https://github.com/thanos-io/thanos/blob/35b8b991177def87ed52dcf10f9b6d87f07282c8/pkg/receive/handler_otlp.go#L39-L75","documentation":"This is an HTTP 429 (Too Many Requests) response sent by Thanos Receive's OTLP ingest handler when the tenant's number of active series has reached the configured limit enforced by the tenant's Limiter. Receive rejects the whole write request rather than accepting partial data, protecting the store from unbounded memory/TSDB growth. The limit is enforced by the local limiter before the request is decoded.","triggerScenarios":"POSTing an OTLP/HTTP metrics export to Receive's /api/v1/otlp/v1/metrics endpoint while the tenant is at or above its active-series limit (queried from the query service by the limiter); the limiter lookup ran but reported the tenant is over limit ('under' == false).","commonSituations":"Cardinality explosion from high-cardinality labels; tenant pushing far more series than provisioned; active-series limit configured too low for the workload; a shard that believes stale series are still active after tenant cardinality dropped.","solutions":["Reduce the number of unique series the tenant exports (drop/rename high-cardinality labels via relabeling at the client or receive config).","Raise the tenant's active series limit in the tenant limit configuration (tenant_limits or the limiter's ActiveSeriesLimit).","Wait for stale/idle series to age out so the tenant's active series count drops below the limit.","Handle HTTP 429 client-side by backing off and retrying later (honor Retry-After header semantics)."],"exampleFix":"// before: unlimited cardinality labels\ncounter.WithLabelValues(userID, region, pod, queue, endpoint)\n// after: bound label cardinality\ncounter.WithLabelValues(region, normalizedEndpoint)","handlingStrategy":"retry","validationCode":"resp, err := http.Post(url, contentType, body)\nif resp.StatusCode == http.StatusTooManyRequests {\n    backoffAndRequeue(body)\n}","typeGuard":null,"tryCatchPattern":"if resp.StatusCode == 429 { /* exponential backoff, then retry */ }","preventionTips":["Monitor tenant active series vs limit and alert at 80%","Apply label-dropping relabeling for high-cardinality labels","Set exporter retry/backoff honoring 429"],"tags":["http","rate-limiting","thanos","monitoring"],"backgroundTag":"rate-limit-exceeded","analyzedSha":"35b8b991177def87ed52dcf10f9b6d87f07282c8","analyzedAt":"2026-09-07T01:49:59.689Z","contentChangedAt":"2026-09-07T01:49:59.689Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}