What causes "internal server error" in thanos-io/thanos?

Increase the write-gate concurrency limit (--receive.write-gate or limiter config) so requests are not queued long enough to hit client timeouts.

thanos-io/thanos · error

internal server error

Error message

internal server error

What it means

This log message accompanies an HTTP 500 response in Handler.receiveOTLPHTTP when writeGate.Start(r.Context()) fails. The write gate is a concurrency limiter limiting simultaneous OTLP/remote-write requests; it fails when the gate's context is cancelled or its internal semaphore cannot be acquired (e.g. server shutting down or context already cancelled). The handler logs "internal server error" and returns the underlying error with status 500.

Solutions

Increase the write-gate concurrency limit (--receive.write-gate or limiter config) so requests are not queued long enough to hit client timeouts.
Raise client-side/ingress timeouts so the request context survives waiting at the gate.
Check server logs for shutdown events — a stopping server closes the gate and rejects in-flight waits.
Retry the OTLP push from the collector with backoff; 500s at the gate are typically transient under load.

Example fix

// collector retry config to survive transient 500s
retry_on_failure:
  enabled: true
  initial_interval: 5s
  max_interval: 30s
  max_elapsed_time: 300s

Defensive patterns

Strategy: retry

Try / catch

// Go: treat 500 at the write gate as retryable
resp, err := client.Do(req)
if err == nil && resp.StatusCode == http.StatusInternalServerError {
	time.Sleep(backoff)
	return retry(req.Clone(ctx))
}

Prevention

Size the write gate above peak concurrent OTLP request concurrency.
Set client timeouts longer than worst-case gate wait time.
Enable collector retry_on_failure with exponential backoff.
Watch server shutdown events; drain exporters before restarting receive.

When it happens

Trigger: Raised when tracing.DoInSpan wraps writeGate.Start(r.Context()) and the returned err is non-nil after the deferred writeGate.Done() — i.e. the request context was cancelled/expired before a gate slot was acquired, or the gate is closed.

Common situations: Client timeout cancels r.Context() while the request waits at the concurrency gate under heavy load; gateway timeout cutting long-queued OTLP requests; server shutdown closing the write gate mid-request; write-gate limit set too low for the ingest rate.

Understand the failure class

HTTP status errors: handling 4xx and 5xx responses — how to handle 4xx and 5xx responses properly.

Related errors

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/6bff0072aef76a0c. Report an issue: GitHub.

Appendix: source

Thrown at pkg/receive/handler_otlp.go:46

	tenant, err := tenancy.GetTenantFromHTTP(r, h.options.TenantHeader, h.options.DefaultTenantID, h.options.TenantField)
	if err != nil {
		level.Error(h.logger).Log("msg", "error getting tenant from HTTP", "err", err)
		http.Error(w, err.Error(), http.StatusBadRequest)
		return
	}

	tLogger := log.With(h.logger, "tenant", tenant)
	span.SetTag("tenant", tenant)

	writeGate := h.Limiter.WriteGate()
	tracing.DoInSpan(r.Context(), "receive_write_gate_ismyturn", func(ctx context.Context) {
		err = writeGate.Start(r.Context())
	})

	defer writeGate.Done()
	if err != nil {
		level.Error(tLogger).Log("err", err, "msg", "internal server error")
		http.Error(w, err.Error(), http.StatusInternalServerError)
		return
	}

	under, err := h.Limiter.HeadSeriesLimiter().isUnderLimit(tenant)
	if err != nil {
		level.Error(tLogger).Log("msg", "error while limiting", "err", err.Error())
	}

	// Fail request fully if tenant has exceeded set limit.
	if !under {
		http.Error(w, "tenant is above active series limit", http.StatusTooManyRequests)
		return
	}

	requestLimiter := h.Limiter.RequestLimiter()
	if r.ContentLength >= 0 {
		if !requestLimiter.AllowSizeBytes(tenant, r.ContentLength) {
			http.Error(w, "write request too large", http.StatusRequestEntityTooLarge)

View on GitHub (pinned to 35b8b99117)