thanos-io/thanos · warning

failed to read body

Error message

failed to read body

What it means

ConfiguredFlags reads the response body with io.ReadAll; if reading fails (connection reset mid-body, truncated response, context cancellation during body read), a plain 'failed to read body' error is returned. Unlike the wrapped request error, this one does not include the underlying cause text beyond the sentinel message.

Solutions

1. Retry the request — this is typically transient network interruption
2. Check logs on the Prometheus/proxy side for connection resets
3. Increase or check proxy/load-balancer idle timeouts
4. Ensure the context isn't canceled prematurely (deadline too tight)

Example fix

// before: single attempt
flags, err := client.ConfiguredFlags(ctx, base)
// after: retry transient read failures
var flags promclient.Flags
err := retry.Do(func() error {
    var e error
    flags, e = client.ConfiguredFlags(ctx, base)
    return e
}, retry.Attempts(3), retry.RetryIf(func(err error) bool {
    return strings.Contains(err.Error(), "failed to read body")
}))

var flags promclient.Flags
err := backoff.Retry(func() error {
    var err error
    flags, err = client.ConfiguredFlags(ctx, u)
    if err != nil && strings.Contains(err.Error(), "failed to read body") {
        return backoff.Permanent(err) // or retry, depending on policy
    }
    return err
}, backoff.WithMaxRetries(backoff.NewExponentialBackOff(), 3))

When it happens

Trigger: io.ReadAll(resp.Body) errors after a successful HTTP response — server closed the connection prematurely, network interruption during body transfer, or the request context was canceled mid-read. Called by an anonymous caller fetching Prometheus flags.

Common situations: Flaky network between Thanos and Prometheus; load balancer idle timeouts cutting the response; Prometheus restarting while serving the request; proxy buffering limits truncating large flag payloads.

Understand the failure class

Background: "failed to read file", EACCES, ENOENT and "could not read <path>" errors: when a program can't read a file from disk — this error's family across 49 libraries.

Related errors

• error starting web server
• failed to validate prometheus flags
• error sending proto response
• send request to
• perform request against

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/371de7cfc0669021. Report an issue: GitHub.

Appendix: source

Thrown at pkg/promclient/promclient.go:312

	u.Path = path.Join(u.Path, "/api/v1/status/flags")

	req, err := http.NewRequest(http.MethodGet, u.String(), nil)
	if err != nil {
		return Flags{}, errors.Wrap(err, "create request")
	}

	span, ctx := tracing.StartSpan(ctx, "/prom_flags HTTP[client]")
	defer span.Finish()

	resp, err := c.Do(req.WithContext(ctx))
	if err != nil {
		return Flags{}, errors.Wrapf(err, "request config against %s", u.String())
	}
	defer runutil.ExhaustCloseWithLogOnErr(c.logger, resp.Body, "query body")

	b, err := io.ReadAll(resp.Body)
	if err != nil {
		return Flags{}, errors.New("failed to read body")
	}

	switch resp.StatusCode {
	case 404:
		return Flags{}, ErrFlagEndpointNotFound
	case 200:
		var d struct {
			Data Flags `json:"data"`
		}

		if err := json.Unmarshal(b, &d); err != nil {
			return Flags{}, errors.Wrapf(err, "unmarshal response: %v", string(b))
		}

		return d.Data, nil
	default:
		return Flags{}, errors.Errorf("got non-200 response code: %v, response: %v", resp.StatusCode, string(b))
	}

View on GitHub (pinned to 35b8b99117)