thanos-io/thanos · error

could not parse replica header

Error message

could not parse replica header

What it means

HTTP 400 returned when the replica header configured on the receive handler (h.options.ReplicaHeader, e.g. THANOS-REPLICA) is present but its value is not a valid unsigned 64-bit integer parseable by strconv.ParseUint. The replica number marks which receive node originally handled the request for deduplication of replicated writes.

Solutions

1. Set the replica header value to a plain non-negative integer (e.g. "0", "1", "2").
2. Fix the proxy/router that injects the header so it forwards the numeric replica index, not the hostname.
3. Ensure the sending receive replica is configured with a matching numeric replica index in its hashring/replica config.
4. Remove the header entirely if the request is not replicated (empty header is treated as non-replicated).

Example fix

// before
req.Header.Set("THANOS-REPLICA", "receive-0.example.com")
// after
req.Header.Set("THANOS-REPLICA", "0")

if v := req.Header.Get("THANOS-REPLICA"); v != "" {
    if _, err := strconv.ParseUint(v, 10, 64); err != nil {
        req.Header.Del("THANOS-REPLICA")
    }
}

func validReplicaHeader(v string) bool { _, err := strconv.ParseUint(v, 10, 64); return err == nil }

if resp.StatusCode == 400 && strings.Contains(respBody, "replica header") { fixForwardingProxyHeader() }

When it happens

Trigger: A client or upstream receive node sets the replica header to a non-numeric value (e.g. a hostname, empty-with-spaces value, or signed/oversized number).

Common situations: Misconfigured load balancer injecting the header with the node name instead of the replica index; hand-rolled forwarding scripts copying the wrong header; version mismatch where a peer sets a differently formatted header.

Understand the failure class

Background: "Invalid ... format", "must be in format X", "does not look like a ..." — invalid argument format errors across CLI tools and libraries — this error's family across 17 libraries.

Related errors

• Admin operations are disabled
• tenant is above active series limit
• write request too large
• error converting OTLP metrics to Prometheus format
• too many timeseries

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/7a64fccf6ab3aa6c. Report an issue: GitHub.

Appendix: source

Thrown at pkg/receive/handler_otlp.go:101

	for _, ts := range metrics {
		totalSamples += len(ts.Samples)
	}

	if !requestLimiter.AllowSeries(tenant, int64(len(metrics))) {
		http.Error(w, "too many timeseries", http.StatusRequestEntityTooLarge)
		return
	}

	if !requestLimiter.AllowSamples(tenant, int64(totalSamples)) {
		http.Error(w, "too many samples", http.StatusRequestEntityTooLarge)
		return
	}

	rep := uint64(0)
	// If the header is empty, we assume the request is not yet replicated.
	if replicaRaw := r.Header.Get(h.options.ReplicaHeader); replicaRaw != "" {
		if rep, err = strconv.ParseUint(replicaRaw, 10, 64); err != nil {
			http.Error(w, "could not parse replica header", http.StatusBadRequest)
			return
		}
	}

	//TODO: (nicolastakashi) Handle metadata in the future.
	wreq := tprompb.WriteRequest{
		Timeseries: metrics,
	}

	// Exit early if the request contained no data. We don't support metadata yet. We also cannot fail here, because
	// this would mean lack of forward compatibility for remote write proto.
	if len(wreq.Timeseries) == 0 {
		// TODO(yeya24): Handle remote write metadata.
		if len(wreq.Metadata) > 0 {
			// TODO(bwplotka): Do we need this error message?
			level.Debug(tLogger).Log("msg", "only metadata from client; metadata ingestion not supported; skipping")
			return
		}

View on GitHub (pinned to 35b8b99117)