thanos-io/thanos · error

expected 2xx response, got

Error message

expected 2xx response, got %d. Body: %v

What it means

req2xx reads the full HTTP response body from a Prometheus API call and throws this error when the status code is not in the 2xx range. It is the generic non-2xx guard used by all client methods (ExternalLabels, QueryInstant, QueryRange, AlertmanagerAlerts, BuildVersion, get2xxResultWithGRPCErrors), carrying the actual status code and response body so the caller can diagnose the server-side failure.

Solutions

1. Inspect the status code and body included in the error to identify the server-side cause
2. Verify the Prometheus/Alertmanager base URL and API path prefix are correct
3. Check authentication/tenancy headers required by the target (e.g. Thanos-Scope, auth bearer)
4. Validate PromQL query syntax before calling QueryInstant/QueryRange
5. Add retry with backoff for transient 5xx responses

Example fix

// before: error surfaced raw, hard to tell 400 from 503
labels, err := client.QueryInstant(ctx, rng, timeout, query)
// after: distinguish non-2xx classes
labels, err := client.QueryInstant(ctx, rng, timeout, query)
if err != nil {
    var terr interface{ StatusCode() int }
    if errors.As(err, &terr) && terr.StatusCode() >= 500 {
        // retry with backoff
    }
    return err
}

if err != nil {
    if strings.Contains(err.Error(), "expected 2xx response, got 5") {
        // transient server error: retry with backoff
    } else if strings.Contains(err.Error(), "got 4") {
        // client error: fail fast, fix request
    }
    return err
}

When it happens

Trigger: Any Client method that goes through req2xx when Prometheus/Alertmanager returns 4xx or 5xx: bad query (400), missing/invalid tenant or auth header (401/403), not-found endpoints (404), Prometheus restarting or overloaded (500/503), gateway timeouts from proxies.

Common situations: Querying a Prometheus that returns '422 unprocessable entity' for a bad PromQL expression; hitting Thanos Query with a wrong tenant header; scraping Alertmanager API with mismatched api version; sidecar behind a proxy that returns 502.

Related errors

• failed to validate prometheus flags
• failed to get prometheus version
• initial external labels query
• ErrFlagEndpointNotFound
• create request

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/5474c50dac781aec. Report an issue: GitHub.

Appendix: source

Thrown at pkg/promclient/promclient.go:144

	if c.userAgent != "" {
		req.Header.Set("User-Agent", c.userAgent)
	}
	if method == http.MethodPost {
		req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
	}

	resp, err := c.Do(req.WithContext(ctx))
	if err != nil {
		return nil, 0, errors.Wrapf(err, "perform %s request against %s", method, u.String())
	}
	defer runutil.ExhaustCloseWithErrCapture(&err, resp.Body, "%s: close body", req.URL.String())

	body, err := io.ReadAll(resp.Body)
	if err != nil {
		return nil, resp.StatusCode, errors.Wrap(err, "read body")
	}
	if resp.StatusCode/100 != 2 {
		return nil, resp.StatusCode, errors.Errorf("expected 2xx response, got %d. Body: %v", resp.StatusCode, string(body))
	}
	return body, resp.StatusCode, nil
}

// IsWALDirAccessible returns no error if WAL dir can be found. This helps to tell
// if we have access to Prometheus TSDB directory.
func IsWALDirAccessible(dir string) error {
	const errMsg = "WAL dir is not accessible. Is this dir a TSDB directory? If yes it is shared with TSDB?"

	f, err := os.Stat(filepath.Join(dir, "wal"))
	if err != nil {
		return errors.Wrap(err, errMsg)
	}

	if !f.IsDir() {
		return errors.New(errMsg)
	}

View on GitHub (pinned to 35b8b99117)