thanos-io/thanos · warning

error getting tenant from HTTP

Error message

error getting tenant from HTTP

What it means

This is the log message (and HTTP 400 response) emitted by Handler.receiveOTLPHTTP when tenancy.GetTenantFromHTTP fails to resolve a tenant from the incoming OTLP/HTTP request. Thanos receive requires a tenant identity (via the tenant header, default tenant ID, or tenant field) to apply limits and store data; if extraction fails the request is rejected as a client error. The error message logged is "error getting tenant from HTTP" with the underlying reason in the err field.

Solutions

1. Configure the OTLP exporter/collector to send the expected tenant header (e.g. X-Scope-OrgID).
2. Align the client's tenant header name with the server's --receive.tenant-header option.
3. Check ingress/proxy configurations (e.g. nginx, Envoy) for header stripping rules and allow the tenant header through.
4. If anonymous writes are acceptable, set a non-empty default tenant via the receive default-tenant flag so extraction never fails.

Example fix

// before: OTLP exporter without tenant header
grpcHeaders: {}
// after
gRPCMetadata:
  - key: "x-scope-orgid"
    value: "my-tenant"

// client side: ensure the tenant header is present before sending OTLP
const tenantHeader = "X-Scope-OrgID" // must match --receive.tenant-header
if req.Header.Get(tenantHeader) == "" {
	req.Header.Set(tenantHeader, tenant)
}

// Go: handle 400 from the receiver
resp, err := http.DefaultClient.Do(req)
if err == nil && resp.StatusCode == http.StatusBadRequest {
	body, _ := io.ReadAll(resp.Body)
	return fmt.Errorf("tenant rejected by receiver (status 400): %s", body)
}

When it happens

Trigger: Raised when tenancy.GetTenantFromHTTP(r, h.options.TenantHeader, h.options.DefaultTenantID, h.options.TenantField) returns an error — typically because the configured tenant header is missing from the request, or the request's tenant value is empty/invalid given the handler's tenancy options.

Common situations: OTLP SDK/collector not configured to send the X-Scope-OrgID (or custom --receive.tenant-header) header; proxy or ingress strips the custom tenant header; client sends the tenant in a different header than the server expects; default tenant ID misconfigured.

Related errors

• internal server error
• Error decoding remote write request
• error converting OTLP metrics to Prometheus format
• tenant name not valid
• query-frontend.org-id-header and…

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/7343a6e7e7c653d0. Report an issue: GitHub.

Appendix: source

Thrown at pkg/receive/handler_otlp.go:31

	"github.com/pkg/errors"
	"github.com/prometheus/prometheus/storage/remote"
	"github.com/thanos-io/thanos/pkg/receive/otlptranslator"
	tprompb "github.com/thanos-io/thanos/pkg/store/storepb/prompb"
	"github.com/thanos-io/thanos/pkg/tenancy"
	"github.com/thanos-io/thanos/pkg/tracing"
	"go.opentelemetry.io/collector/pdata/pmetric"
)

func (h *Handler) receiveOTLPHTTP(w http.ResponseWriter, r *http.Request) {
	var err error
	span, ctx := tracing.StartSpan(r.Context(), "receive_otlp_http")
	span.SetTag("receiver.mode", string(h.receiverMode))
	defer span.Finish()

	tenant, err := tenancy.GetTenantFromHTTP(r, h.options.TenantHeader, h.options.DefaultTenantID, h.options.TenantField)
	if err != nil {
		level.Error(h.logger).Log("msg", "error getting tenant from HTTP", "err", err)
		http.Error(w, err.Error(), http.StatusBadRequest)
		return
	}

	tLogger := log.With(h.logger, "tenant", tenant)
	span.SetTag("tenant", tenant)

	writeGate := h.Limiter.WriteGate()
	tracing.DoInSpan(r.Context(), "receive_write_gate_ismyturn", func(ctx context.Context) {
		err = writeGate.Start(r.Context())
	})

	defer writeGate.Done()
	if err != nil {
		level.Error(tLogger).Log("err", err, "msg", "internal server error")
		http.Error(w, err.Error(), http.StatusInternalServerError)
		return
	}

View on GitHub (pinned to 35b8b99117)