thanos-io/thanos · error

Error decoding remote write request

Error message

Error decoding remote write request

What it means

This log message accompanies an HTTP 400 returned when remote.DecodeOTLPWriteRequest fails to parse the incoming OTLP/HTTP payload. The handler logs the decode error server-side and returns the raw error text to the client. It means the request body is not a valid protobuf-encoded ExportMetricsServiceRequest (with allowed content-type/encoding).

Solutions

1. Ensure the client sends Content-Type: application/x-protobuf with a valid protobuf-encoded OTLP ExportMetricsServiceRequest.
2. If sending JSON, configure the client for OTLP/HTTP JSON and verify receive accepts that content type.
3. Check Content-Encoding: gzip is set correctly and the body is compressed exactly once.
4. Read the returned error text in the 400 response; it names the exact decode failure.
5. Upgrade/downgrade the OTLP client library to match the receiver's prometheus/thanos OTLP support.

Example fix

// before: JSON body without proper content type
req.Header.Set("Content-Type", "application/json")
// after
req.Header.Set("Content-Type", "application/x-protobuf")
req.Header.Set("Content-Encoding", "gzip")

// client side: ensure protobuf encoding
req.Header.Set("Content-Type", "application/x-protobuf")
// optionally verify body parses before send
proto.Unmarshal(payload, &colmetricpb.ExportMetricsServiceRequest{})

if resp.StatusCode == 400 { log.Fatalf("OTLP decode rejected: %s", resp.Body) }

When it happens

Trigger: POSTing a body to /api/v1/otlp/v1/metrics that is not valid OTLP protobuf (wrong content-type, e.g. JSON when protobuf expected, or gzipped body without Content-Encoding: gzip, truncated body, wrong proto version).

Common situations: Client using OTLP/HTTP JSON instead of protobuf; double-compression or missing gzip header; network proxy rewriting the body; mismatched OTLP protocol versions between client and receive.

Understand the failure class

Background: "cannot parse invalid wire-format data", "cannot unmarshal", "failed unmarshalling": protobuf unmarshal errors explained — this error's family across 10 libraries.

Related errors

• error marshaling proto response
• error sending proto response
• error getting tenant from HTTP
• internal server error
• error converting OTLP metrics to Prometheus format

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/35b31684145f4a1a. Report an issue: GitHub.

Appendix: source

Thrown at pkg/receive/handler_otlp.go:72

	// Fail request fully if tenant has exceeded set limit.
	if !under {
		http.Error(w, "tenant is above active series limit", http.StatusTooManyRequests)
		return
	}

	requestLimiter := h.Limiter.RequestLimiter()
	if r.ContentLength >= 0 {
		if !requestLimiter.AllowSizeBytes(tenant, r.ContentLength) {
			http.Error(w, "write request too large", http.StatusRequestEntityTooLarge)
			return
		}
	}

	req, err := remote.DecodeOTLPWriteRequest(r)
	if err != nil {
		level.Error(h.logger).Log("msg", "Error decoding remote write request", "err", err.Error())
		http.Error(w, err.Error(), http.StatusBadRequest)
		return
	}

	metrics, _, err := h.convertToPrometheusFormat(ctx, req.Metrics())
	if err != nil {
		http.Error(w, err.Error(), http.StatusBadRequest)
		return
	}

	totalSamples := 0
	for _, ts := range metrics {
		totalSamples += len(ts.Samples)
	}

	if !requestLimiter.AllowSeries(tenant, int64(len(metrics))) {
		http.Error(w, "too many timeseries", http.StatusRequestEntityTooLarge)
		return
	}

View on GitHub (pinned to 35b8b99117)