thanos-io/thanos · error

request failed with code

Error message

request failed with code %s; msg %s

What it means

When the remote read endpoint returns a non-2xx status, startPromRemoteRead reads the body (best effort) and returns errors.Errorf("request failed with code %s; msg %s", presp.Status, string(b)). The error carries the HTTP status and whatever error text the server returned.

Solutions

1. Read the embedded msg in the error — it contains the server's explanation of the non-2xx status
2. If the status mentions unsupported response types, restrict remoteReadAcceptableResponses to non-streamed (or upgrade Prometheus to >= 2.13)
3. Fix authentication at any reverse proxy in front of Prometheus (headers, client certs, tokens)
4. Check Prometheus server logs at the same timestamp for the corresponding server-side error

Example fix

// before: request streamed chunks from an old Prometheus
remoteReadAcceptableResponses = []prompb.ReadRequest_ResponseType{
    prompb.ReadRequest_STREAMED_XOR_CHUNKS,
}
// after: fall back to sampled responses
remoteReadAcceptableResponses = []prompb.ReadRequest_ResponseType{
    prompb.ReadRequest_SAMPLES,
}

healthResp, err := http.Get(baseURL + "/-/healthy")
if err != nil || healthResp.StatusCode != 200 { return fmt.Errorf("prometheus unhealthy: status %d", statusCode(healthResp)) }

resp, err := store.Series(ctx, req)
if err != nil && strings.Contains(err.Error(), "request failed with code") {
    // parse embedded status; if it indicates unsupported streamed chunks, retry with SAMPLES-only response type
}

When it happens

Trigger: Prometheus returns 4xx/5xx for /api/v1/read — e.g. 400 for an invalid query, 403 from an auth proxy, 422/500 for unsupported response types (streamed chunks requested of an old Prometheus), or 502 from a broken proxy.

Common situations: Requesting AcceptResponseTypes STREAMED_XOR_CHUNKS against a Prometheus < 2.13 that rejects it; auth proxy (oauth2_proxy/ForwardAuth) rejecting the request; query outside retention returning 4xx; rate limiting 429 at the proxy.

Understand the failure class

Background: "API error: {status}" and "HTTP 401/403/404/429/5xx" errors: non-2xx HTTP responses explained — this error's family across 27 libraries.

Related errors

• query Prometheus
• not supported remote read content type
• copy response
• failed to validate prometheus flags
• failed to get prometheus version

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/35e60ff2a60c41a4. Report an issue: GitHub.

Appendix: source

Thrown at pkg/store/prometheus.go:490

		return nil, errors.Wrap(err, "unable to create request")
	}
	preq.Header.Add("Content-Encoding", "snappy")
	preq.Header.Set("Content-Type", "application/x-stream-protobuf")
	preq.Header.Set("X-Prometheus-Remote-Read-Version", "0.1.0")

	preq.Header.Set("User-Agent", clientconfig.ThanosUserAgent)
	presp, err = p.client.Do(preq.WithContext(ctx))
	if err != nil {
		return nil, errors.Wrap(err, "send request")
	}
	if presp.StatusCode/100 != 2 {
		// Best effort read.
		b, err := io.ReadAll(presp.Body)
		if err != nil {
			level.Error(p.logger).Log("msg", "failed to read response from non 2XX remote read request", "err", err)
		}
		_ = presp.Body.Close()
		return nil, errors.Errorf("request failed with code %s; msg %s", presp.Status, string(b))
	}

	return presp, nil
}

// matchesExternalLabels returns false if given matchers are not matching external labels.
// If true, matchesExternalLabels also returns Prometheus matchers without those matching external labels.
func matchesExternalLabels(ms []storepb.LabelMatcher, externalLabels labels.Labels, cache storecache.MatchersCache) (bool, []*labels.Matcher, error) {
	var (
		tms []*labels.Matcher
		err error
	)

	tms, err = storecache.MatchersToPromMatchersCached(cache, ms...)
	if err != nil {
		return false, nil, err
	}

View on GitHub (pinned to 35b8b99117)