thanos-io/thanos · critical

err.Error()

Error message

err.Error()

What it means

This is the generic internal-server-error path (HTTP 500) of the OTLP ingest handler: after h.writer.Write / forwarding fails with an error that is not errBadReplica or another recognized case, the handler logs 'internal server error' and returns err.Error() with status 500. It surfaces any write/forward failure from the underlying store or replication fan-out.

Solutions

1. Check the receive server logs — the log line includes the underlying err with full detail.
2. Verify storage health: disk space, TSDB head block, and storage.tsdb.path writability.
3. Check network connectivity and DNS between receive nodes for forwarding failures.
4. Confirm all receive nodes share a consistent hashring configuration.
5. Retry the write; for forwarding failures receive retries internally, but persistent 500s indicate an infrastructure problem.

Example fix

# diagnose
kubectl logs -l app=thanos-receive | grep 'internal server error'
# check disk
df -h /var/thanos/store
# verify peers
thanos tools bucket verify 2>/dev/null || curl -s http://receive:10902/-/ready

resp, err := http.Post(url, ct, body)
if err != nil || resp.StatusCode >= 500 { scheduleRetry(body) }

if resp.StatusCode >= 500 { backoff.Retry(func() error { return resend(body) }) } else { failPermanent(resp) }

When it happens

Trigger: Write to the local TSDB head failed (storage errors, out-of-order/duplicate handling disabled), or forwarding to other receive nodes failed (hashring lookup failure, all replicas down, network errors) — anything not classified as a bad replica.

Common situations: TSDB storage full or corrupted; remote receive peers unreachable due to network policies; hashring config mismatch between receive nodes; head block compaction issues.

Understand the failure class

Background: "API error: {status}" and "HTTP 401/403/404/429/5xx" errors: non-2xx HTTP responses explained — this error's family across 27 libraries.

Related errors

• critical error detected
• start remote write agent db
• open TSDB
• Admin operations are disabled
• tenant is above active series limit

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/670b4e2df80ce2d8. Report an issue: GitHub.

Appendix: source

Thrown at pkg/receive/handler_otlp.go:153

			wreq:   &wreq,
		},
	})
	if err != nil {
		level.Debug(tLogger).Log("msg", "failed to handle request", "err", err.Error())
		switch errors.Cause(err) {
		case errNotReady:
			responseStatusCode = http.StatusServiceUnavailable
		case errUnavailable:
			responseStatusCode = http.StatusServiceUnavailable
		case errConflict:
			responseStatusCode = http.StatusConflict
		case errBadReplica:
			responseStatusCode = http.StatusBadRequest
		default:
			level.Error(tLogger).Log("err", err, "msg", "internal server error")
			responseStatusCode = http.StatusInternalServerError
		}
		http.Error(w, err.Error(), responseStatusCode)
	}

	for tenant, stats := range tenantStats {
		h.writeTimeseriesTotal.WithLabelValues(strconv.Itoa(responseStatusCode), tenant).Observe(float64(stats.timeseries))
		h.writeSamplesTotal.WithLabelValues(strconv.Itoa(responseStatusCode), tenant).Observe(float64(stats.totalSamples))
	}

}

func (h *Handler) convertToPrometheusFormat(ctx context.Context, pmetrics pmetric.Metrics) ([]tprompb.TimeSeries, []tprompb.MetricMetadata, error) {
	converter := otlptranslator.NewPrometheusConverter()
	settings := otlptranslator.Settings{
		AddMetricSuffixes:         true,
		DisableTargetInfo:         !h.options.OtlpEnableTargetInfo,
		PromoteResourceAttributes: h.options.OtlpResourceAttributes,
	}

	annots, err := converter.FromMetrics(ctx, pmetrics, settings)

View on GitHub (pinned to 35b8b99117)