thanos-io/thanos · error

copy response

Error message

copy response

What it means

fetchSampledResponse reads the entire sampled remote-read body with io.Copy into a pooled buffer; failure is wrapped as 'copy response'. This covers reading the HTTP response body from the Prometheus sampled (x-protobuf) remote read endpoint.

Solutions

1. Check the wrapped cause for reset vs timeout and fix the underlying network/timeout setting.
2. Reduce the time range or step to shrink the sampled response size.
3. Raise proxy/LB response timeouts for long remote-read requests.
4. Prefer streamed chunked remote read (supported content type) for large series sets.
5. Retry the query; transient body-read failures often succeed on retry.

data, err := s.fetchSampledResponse(ctx, resp, ...)
if err != nil {
    if strings.Contains(err.Error(), "copy response") {
        // body read interrupted: retry; check errors.Unwrap for unexpected EOF/reset
    }
    return err
}

When it happens

Trigger: Calling Series on a PrometheusStore using sampled remote read when the response body cannot be fully read: connection reset, unexpected EOF, timeouts mid-body, or server aborting the response.

Common situations: Large sampled responses cut off by proxy timeouts; network flakiness; Prometheus closing connections under load; keep-alive races.

Related errors

• query Prometheus
• failed to validate prometheus flags
• perform request against
• request config against
• read query instant response

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/f612d40799512c85. Report an issue: GitHub.

Appendix: source

Thrown at pkg/store/prometheus.go:401

func (s *BytesCounter) Read(p []byte) (n int, err error) {
	n, err = s.ReadCloser.Read(p)
	s.bytesCount += n
	return n, err
}

func (s *BytesCounter) BytesCount() int {
	return s.bytesCount
}

func (p *PrometheusStore) fetchSampledResponse(ctx context.Context, resp *http.Response) (_ *prompb.ReadResponse, err error) {
	defer runutil.ExhaustCloseWithLogOnErr(p.logger, resp.Body, "prom series request body")

	b := p.getBuffer()
	buf := bytes.NewBuffer(*b)
	defer p.putBuffer(b)
	if _, err := io.Copy(buf, resp.Body); err != nil {
		return nil, errors.Wrap(err, "copy response")
	}

	sb := p.getBuffer()
	var decomp []byte
	tracing.DoInSpan(ctx, "decompress_response", func(ctx context.Context) {
		decomp, err = snappy.Decode(*sb, buf.Bytes())
	})
	defer p.putBuffer(sb)
	if err != nil {
		return nil, errors.Wrap(err, "decompress response")
	}

	var data prompb.ReadResponse
	tracing.DoInSpan(ctx, "unmarshal_response", func(ctx context.Context) {
		err = proto.Unmarshal(decomp, &data)
	})
	if err != nil {
		return nil, errors.Wrap(err, "unmarshal response")

View on GitHub (pinned to 35b8b99117)