thanos-io/thanos · error

error converting OTLP metrics to Prometheus format

Error message

error converting OTLP metrics to Prometheus format

What it means

HTTP 400 returned when h.convertToPrometheusFormat fails to translate the decoded OTLP metrics into Prometheus time series. The error text is passed straight to the client, so this is usually an issue with the metric data itself (invalid names/labels, out-of-range values, unsupported metric types) rather than transport.

Solutions

1. Read the 400 response body; it contains the exact conversion error.
2. Fix instrument names at the SDK to be valid Prometheus identifiers (use the SDK's name normalization / validation).
3. Check the SDK isn't emitting metrics with empty names or NaN/invalid values.
4. Upgrade Thanos/Prometheus to a build supporting the metric types you export (e.g. exponential histograms).
5. Verify the request context isn't being cancelled upstream (client timeouts mid-flight).

Example fix

// before: invalid instrument name
meter.CreateFloatCounter("http.requests!total")
// after: valid Prometheus-safe name
meter.CreateFloatCounter("http_requests_total")

// validate metric names before export
func validPromName(s string) bool {
  for _, r := range s {
    if !(r=='_'||r==':'||(r>='a'&&r<='z')||(r>='A'&&r<='Z')||(r>='0'&&r<='9')) { return false }
  }
  return s != ""
}

func isValidMetric(m Metric) bool { return m != nil && validPromName(m.Name()) }

if resp.StatusCode == 400 { inspectBody(errText); fixInstrumentNames() }

When it happens

Trigger: An OTLP ExportMetricsServiceRequest that decodes fine but contains data prometheus's otlp translator rejects: empty/invalid metric or unit names, histogram/exponential-histogram data the converter cannot handle, or a context cancellation/limiter error during conversion.

Common situations: Instruments named with characters invalid in Prometheus metric names without normalization; empty metric names from misconfigured SDKs; unsupported metric types sent by newer SDKs to older receive builds.

Understand the failure class

Background: "Must be a positive integer", "Invalid value", "Unsupported": the invalid-argument-value error family, when a library rejects the value you pass — this error's family across 35 libraries.

Related errors

• Admin operations are disabled
• convert matchers
• converting prom matchers to storepb matchers
• error getting tenant from HTTP
• internal server error

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/f45d11043a5ee151. Report an issue: GitHub.

Appendix: source

Thrown at pkg/receive/handler_otlp.go:78

	requestLimiter := h.Limiter.RequestLimiter()
	if r.ContentLength >= 0 {
		if !requestLimiter.AllowSizeBytes(tenant, r.ContentLength) {
			http.Error(w, "write request too large", http.StatusRequestEntityTooLarge)
			return
		}
	}

	req, err := remote.DecodeOTLPWriteRequest(r)
	if err != nil {
		level.Error(h.logger).Log("msg", "Error decoding remote write request", "err", err.Error())
		http.Error(w, err.Error(), http.StatusBadRequest)
		return
	}

	metrics, _, err := h.convertToPrometheusFormat(ctx, req.Metrics())
	if err != nil {
		http.Error(w, err.Error(), http.StatusBadRequest)
		return
	}

	totalSamples := 0
	for _, ts := range metrics {
		totalSamples += len(ts.Samples)
	}

	if !requestLimiter.AllowSeries(tenant, int64(len(metrics))) {
		http.Error(w, "too many timeseries", http.StatusRequestEntityTooLarge)
		return
	}

	if !requestLimiter.AllowSamples(tenant, int64(totalSamples)) {
		http.Error(w, "too many samples", http.StatusRequestEntityTooLarge)
		return
	}

View on GitHub (pinned to 35b8b99117)