{"record":{"id":"6bff0072aef76a0c","repo":"thanos-io/thanos","slug":"internal-server-error","errorCode":null,"errorMessage":"internal server error","messagePattern":"internal server error","errorType":"http","errorClass":null,"httpStatus":500,"severity":"error","filePath":"pkg/receive/handler_otlp.go","lineNumber":46,"sourceCode":"\ttenant, err := tenancy.GetTenantFromHTTP(r, h.options.TenantHeader, h.options.DefaultTenantID, h.options.TenantField)\n\tif err != nil {\n\t\tlevel.Error(h.logger).Log(\"msg\", \"error getting tenant from HTTP\", \"err\", err)\n\t\thttp.Error(w, err.Error(), http.StatusBadRequest)\n\t\treturn\n\t}\n\n\ttLogger := log.With(h.logger, \"tenant\", tenant)\n\tspan.SetTag(\"tenant\", tenant)\n\n\twriteGate := h.Limiter.WriteGate()\n\ttracing.DoInSpan(r.Context(), \"receive_write_gate_ismyturn\", func(ctx context.Context) {\n\t\terr = writeGate.Start(r.Context())\n\t})\n\n\tdefer writeGate.Done()\n\tif err != nil {\n\t\tlevel.Error(tLogger).Log(\"err\", err, \"msg\", \"internal server error\")\n\t\thttp.Error(w, err.Error(), http.StatusInternalServerError)\n\t\treturn\n\t}\n\n\tunder, err := h.Limiter.HeadSeriesLimiter().isUnderLimit(tenant)\n\tif err != nil {\n\t\tlevel.Error(tLogger).Log(\"msg\", \"error while limiting\", \"err\", err.Error())\n\t}\n\n\t// Fail request fully if tenant has exceeded set limit.\n\tif !under {\n\t\thttp.Error(w, \"tenant is above active series limit\", http.StatusTooManyRequests)\n\t\treturn\n\t}\n\n\trequestLimiter := h.Limiter.RequestLimiter()\n\tif r.ContentLength >= 0 {\n\t\tif !requestLimiter.AllowSizeBytes(tenant, r.ContentLength) {\n\t\t\thttp.Error(w, \"write request too large\", http.StatusRequestEntityTooLarge)","sourceCodeStart":28,"sourceCodeEnd":64,"githubUrl":"https://github.com/thanos-io/thanos/blob/35b8b991177def87ed52dcf10f9b6d87f07282c8/pkg/receive/handler_otlp.go#L28-L64","documentation":"This log message accompanies an HTTP 500 response in Handler.receiveOTLPHTTP when writeGate.Start(r.Context()) fails. The write gate is a concurrency limiter limiting simultaneous OTLP/remote-write requests; it fails when the gate's context is cancelled or its internal semaphore cannot be acquired (e.g. server shutting down or context already cancelled). The handler logs \"internal server error\" and returns the underlying error with status 500.","triggerScenarios":"Raised when tracing.DoInSpan wraps writeGate.Start(r.Context()) and the returned err is non-nil after the deferred writeGate.Done() — i.e. the request context was cancelled/expired before a gate slot was acquired, or the gate is closed.","commonSituations":"Client timeout cancels r.Context() while the request waits at the concurrency gate under heavy load; gateway timeout cutting long-queued OTLP requests; server shutdown closing the write gate mid-request; write-gate limit set too low for the ingest rate.","solutions":["Increase the write-gate concurrency limit (--receive.write-gate or limiter config) so requests are not queued long enough to hit client timeouts.","Raise client-side/ingress timeouts so the request context survives waiting at the gate.","Check server logs for shutdown events — a stopping server closes the gate and rejects in-flight waits.","Retry the OTLP push from the collector with backoff; 500s at the gate are typically transient under load."],"exampleFix":"// collector retry config to survive transient 500s\nretry_on_failure:\n  enabled: true\n  initial_interval: 5s\n  max_interval: 30s\n  max_elapsed_time: 300s","handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"// Go: treat 500 at the write gate as retryable\nresp, err := client.Do(req)\nif err == nil && resp.StatusCode == http.StatusInternalServerError {\n\ttime.Sleep(backoff)\n\treturn retry(req.Clone(ctx))\n}","preventionTips":["Size the write gate above peak concurrent OTLP request concurrency.","Set client timeouts longer than worst-case gate wait time.","Enable collector retry_on_failure with exponential backoff.","Watch server shutdown events; drain exporters before restarting receive."],"tags":["http","internal-server-error","otlp","concurrency","write-gate"],"backgroundTag":"http-500-internal-server-error","analyzedSha":"35b8b991177def87ed52dcf10f9b6d87f07282c8","analyzedAt":"2026-09-07T01:49:59.689Z","contentChangedAt":"2026-09-07T01:49:59.689Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}