What causes "decompress response" in thanos-io/thanos?

Log/inspect the raw response body to see if it's an error page instead of snappy protobuf data.

decompress response

Error message

decompress response

What it means

After copying the response, fetchSampledResponse snappy-decodes the raw bytes into a pooled buffer; a decode failure is wrapped as 'decompress response'. The payload received was not valid snappy-compressed remote-read data.

Solutions

Log/inspect the raw response body to see if it's an error page instead of snappy protobuf data.
Fix auth/proxy layers that return non-snappy error bodies for remote-read requests.
Check for truncation (compare Content-Length with bytes read) and raise proxy timeouts.
Verify the target endpoint actually supports snappy-compressed sampled remote read.

Defensive patterns

Strategy: validation

Validate before calling

body, _ := io.ReadAll(resp.Body)
if len(body) > 0 && body[0] == '<' || bytes.HasPrefix(body, []byte{"{"})) {
    return fmt.Errorf("remote read returned non-snappy payload: %s", truncate(body, 200))
}

Try / catch

data, err := s.fetchSampledResponse(ctx, resp, ...)
if err != nil {
    if strings.Contains(err.Error(), "decompress response") {
        // payload not snappy: inspect raw body for error pages; fix auth/proxy layer
    }
    return err
}

Prevention

Ensure auth middlewares return proper HTTP error status rather than 200 with an error body.
Check for truncating proxies on the remote-read path.
Confirm the endpoint snappy-encodes sampled remote read responses.
Capture and log response bodies on decompression failure for diagnosis.

When it happens

Trigger: The bytes returned by the remote-read endpoint are not valid snappy data — e.g. an error page (HTML/JSON) sent with a protobuf content type, a truncated body, or a corrupted response.

Common situations: Auth/proxy layers returning error bodies while status looks OK; truncated responses cut by proxies; pointing at an endpoint that does not snappy-encode remote read responses as expected.

Related errors

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/0c3f3ff68b3150aa. Report an issue: GitHub.

Appendix: source

Thrown at pkg/store/prometheus.go:411

func (p *PrometheusStore) fetchSampledResponse(ctx context.Context, resp *http.Response) (_ *prompb.ReadResponse, err error) {
	defer runutil.ExhaustCloseWithLogOnErr(p.logger, resp.Body, "prom series request body")

	b := p.getBuffer()
	buf := bytes.NewBuffer(*b)
	defer p.putBuffer(b)
	if _, err := io.Copy(buf, resp.Body); err != nil {
		return nil, errors.Wrap(err, "copy response")
	}

	sb := p.getBuffer()
	var decomp []byte
	tracing.DoInSpan(ctx, "decompress_response", func(ctx context.Context) {
		decomp, err = snappy.Decode(*sb, buf.Bytes())
	})
	defer p.putBuffer(sb)
	if err != nil {
		return nil, errors.Wrap(err, "decompress response")
	}

	var data prompb.ReadResponse
	tracing.DoInSpan(ctx, "unmarshal_response", func(ctx context.Context) {
		err = proto.Unmarshal(decomp, &data)
	})
	if err != nil {
		return nil, errors.Wrap(err, "unmarshal response")
	}
	if len(data.Results) != 1 {
		return nil, errors.Errorf("unexpected result size %d", len(data.Results))
	}

	for _, ts := range data.Results[0].Timeseries {
		labelpb.ReAllocZLabelsStrings(&ts.Labels)
	}

	return &data, nil

View on GitHub (pinned to 35b8b99117)