thanos-io/thanos · error

unmarshal response

Error message

unmarshal response

What it means

PrometheusStore's fetchSampledResponse decompresses a snappy-encoded remote-read response and proto.Unmarshals it into prompb.ReadResponse. If the protobuf payload is corrupt, truncated, or not a valid ReadResponse, the wrapped error "unmarshal response" is returned. It indicates the upstream Prometheus endpoint returned bytes that could not be parsed as the expected remote-read protobuf.

Solutions

1. Verify the store's base URL points at a Prometheus instance supporting the remote read API (/api/v1/read) and that the response Content-Encoding (snappy) is not stripped by proxies
2. Check Prometheus version compatibility with the X-Prometheus-Remote-Read-Version header and AcceptedResponseTypes (STREAMED_XOR_CHUNKS support)
3. Capture the raw response (curl with Accept-Encoding/Content-Encoding headers) and confirm it is snappy-compressed protobuf
4. Confirm request timeouts / size limits (MaxBytesReader) are not truncating the body before unmarshal

Example fix

// before: trusting any 2xx body
presp, err = p.client.Do(preq.WithContext(ctx))
if err != nil {
    return nil, errors.Wrap(err, "send request")
}
// after: also validate response encoding before unmarshal
if enc := presp.Header.Get("Content-Encoding"); enc != "snappy" {
    return nil, errors.Errorf("unexpected response encoding %q", enc)
}

if !strings.Contains(resp.Header.Get("Content-Type"), "protobuf") { return fmt.Errorf("unexpected content type %q", resp.Header.Get("Content-Type")) }

series, err := store.Series(ctx, req)
if err != nil && strings.Contains(err.Error(), "unmarshal response") {
    // treat as upstream protocol issue: check prometheus version/encoding, retry with different response types
}

When it happens

Trigger: Calling Series on a PrometheusStore whose remote /api/v1/read endpoint returns malformed, truncated, or non-protobuf data (e.g. an HTML error page that bypassed the status-code check, a proxy mangling the body, or a Prometheus version responding with an unexpected encoding).

Common situations: Reverse proxy or service mesh rewriting/compressing the response body; hitting the wrong URL (query API instead of read API); Prometheus remote-read returning chunked/streamed encoding the client doesn't expect; snappy compression mismatch between client and server.

Understand the failure class

Background: "cannot parse invalid wire-format data", "cannot unmarshal", "failed unmarshalling": protobuf unmarshal errors explained — this error's family across 10 libraries.

Related errors

• next proto
• marshal read request
• unmarshal response
• unmarshal query instant response
• unmarshal query range response

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/b1445ddce98869d7. Report an issue: GitHub.

Appendix: source

Thrown at pkg/store/prometheus.go:419

		return nil, errors.Wrap(err, "copy response")
	}

	sb := p.getBuffer()
	var decomp []byte
	tracing.DoInSpan(ctx, "decompress_response", func(ctx context.Context) {
		decomp, err = snappy.Decode(*sb, buf.Bytes())
	})
	defer p.putBuffer(sb)
	if err != nil {
		return nil, errors.Wrap(err, "decompress response")
	}

	var data prompb.ReadResponse
	tracing.DoInSpan(ctx, "unmarshal_response", func(ctx context.Context) {
		err = proto.Unmarshal(decomp, &data)
	})
	if err != nil {
		return nil, errors.Wrap(err, "unmarshal response")
	}
	if len(data.Results) != 1 {
		return nil, errors.Errorf("unexpected result size %d", len(data.Results))
	}

	for _, ts := range data.Results[0].Timeseries {
		labelpb.ReAllocZLabelsStrings(&ts.Labels)
	}

	return &data, nil
}

func (p *PrometheusStore) chunkSamples(series *prompb.TimeSeries, maxSamplesPerChunk int, calculateChecksums bool) (chks []storepb.AggrChunk, err error) {
	samples := series.Samples
	hasher := hashPool.Get().(hash.Hash64)
	defer hashPool.Put(hasher)

	for len(samples) > 0 {

View on GitHub (pinned to 35b8b99117)