thanos-io/thanos · error

read body

Error message

read body

What it means

req2xx successfully received an HTTP response but io.ReadAll(resp.Body) failed while reading it, wrapped as "read body". This means the connection broke or timed out mid-response, so the body is incomplete.

Solutions

1. Retry the request — mid-body failures are usually transient
2. Reduce response size (narrow the query time range, add step/limit)
3. Increase LB/proxy idle and read timeouts to exceed worst-case query duration
4. Check Prometheus and proxy logs for restarts or errors during the request window

Example fix

// before
body, err := client.QueryRange(ctx, apiCtx, ...)
// after with retry
err := runutil.Retry(2*time.Second, ctx.Done(), func() error {
    _, err = client.QueryRange(ctx, apiCtx, ...)
    if err != nil && strings.Contains(err.Error(), "read body") {
        return err // transient mid-body read failure: retry
    }
    return runutil.StopRetry(err)
})

body, _, err := client.QueryInstant(ctx, apiCtx, nil)
if err != nil && strings.Contains(err.Error(), "read body") {
    // transient mid-response disconnect: retry with backoff
    return retryWithBackoff(ctx)
}

When it happens

Trigger: Prometheus closes the connection mid-transfer (restart, LB idle timeout, proxy error); large query results truncated by network interruption; context deadline hits while streaming the body.

Common situations: Big QueryRange results over flaky networks; ingress/LB response timeouts shorter than query duration; Prometheus OOM-killed while serving a heavy response.

Related errors

• error starting web server
• failed to validate prometheus flags
• error sending proto response
• send request to
• read meta file

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/77c84b08e8bdb437. Report an issue: GitHub.

Appendix: source

Thrown at pkg/promclient/promclient.go:141

		req.Header = headers
	}

	if c.userAgent != "" {
		req.Header.Set("User-Agent", c.userAgent)
	}
	if method == http.MethodPost {
		req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
	}

	resp, err := c.Do(req.WithContext(ctx))
	if err != nil {
		return nil, 0, errors.Wrapf(err, "perform %s request against %s", method, u.String())
	}
	defer runutil.ExhaustCloseWithErrCapture(&err, resp.Body, "%s: close body", req.URL.String())

	body, err := io.ReadAll(resp.Body)
	if err != nil {
		return nil, resp.StatusCode, errors.Wrap(err, "read body")
	}
	if resp.StatusCode/100 != 2 {
		return nil, resp.StatusCode, errors.Errorf("expected 2xx response, got %d. Body: %v", resp.StatusCode, string(body))
	}
	return body, resp.StatusCode, nil
}

// IsWALDirAccessible returns no error if WAL dir can be found. This helps to tell
// if we have access to Prometheus TSDB directory.
func IsWALDirAccessible(dir string) error {
	const errMsg = "WAL dir is not accessible. Is this dir a TSDB directory? If yes it is shared with TSDB?"

	f, err := os.Stat(filepath.Join(dir, "wal"))
	if err != nil {
		return errors.Wrap(err, errMsg)
	}

	if !f.IsDir() {

View on GitHub (pinned to 35b8b99117)