thanos-io/thanos · error

next proto

Error message

next proto

What it means

handleStreamedPrometheusResponse iterates a streamed chunked remote-read response calling stream.NextProto for each frame. Any decode/transport error other than io.EOF is wrapped as 'next proto', meaning a frame of the ChunkedReadResponse stream failed to be read or unmarshaled.

Solutions

1. Check the wrapped cause (%w) — connection reset vs unmarshal error point to network vs data issues.
2. Inspect Prometheus server logs for restarts/errors during the query window.
3. Reduce query time range/scope to shrink the streamed response, or shard the query.
4. Check intermediary proxies/LBs for idle or response-size timeouts on streaming connections and raise them.
5. Retry the query; transient network resets are common on long streams.

if err := s.Series(ctx, req); err != nil {
    if strings.Contains(err.Error(), "next proto") {
        // transient stream break: retry with backoff; narrow the time range if persistent
    }
    return err
}

When it happens

Trigger: During a Series call with streamed chunked remote read: connection reset mid-stream, truncated/garbled frame, server closing the stream unexpectedly, or a payload failing protobuf decode into prompb.ChunkedReadResponse.

Common situations: Load balancers/idle-timeout killing long streaming responses; Prometheus restarting or OOMing mid-query; proxies buffering/breaking chunked transfer; network instability on large queries.

Related errors

• query Prometheus
• copy response
• unmarshal response
• marshal read request
• failed to validate prometheus flags

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/3eca69fc407a1f8a. Report an issue: GitHub.

Appendix: source

Thrown at pkg/store/prometheus.go:313

	var data = p.getBuffer()
	defer p.putBuffer(data)

	bodySizer := NewBytesRead(httpResp.Body)
	seriesStats := &storepb.SeriesStatsCounter{}

	// TODO(bwplotka): Put read limit as a flag.
	stream := remote.NewChunkedReader(bodySizer, config.DefaultChunkedReadLimit, *data)
	hasher := hashPool.Get().(hash.Hash64)
	defer hashPool.Put(hasher)
	for {
		res := &prompb.ChunkedReadResponse{}
		err := stream.NextProto(res)
		if err == io.EOF {
			break
		}
		if err != nil {
			return errors.Wrap(err, "next proto")
		}

		if len(res.ChunkedSeries) != 1 {
			level.Warn(p.logger).Log("msg", "Prometheus ReadRequest_STREAMED_XOR_CHUNKS returned non 1 series in frame", "series", len(res.ChunkedSeries))
		}

		framesNum++
		for _, series := range res.ChunkedSeries {
			// MergeLabels() prefers local labels over external labels but we prefer
			// external labels hence we need to do this:
			// https://github.com/prometheus/prometheus/blob/3f6f5d3357e232abe53f1775f893fdf8f842712c/storage/remote/codec.go#L210.
			completeLabelset := rmLabels(labelpb.ExtendSortedLabels(labelpb.ZLabelsToPromLabels(series.Labels), extLset), extLsetToRemove)
			if !shardMatcher.MatchesLabels(completeLabelset) {
				continue
			}

			seriesStats.CountSeries(series.Labels)
			thanosChks := make([]storepb.AggrChunk, len(series.Chunks))

View on GitHub (pinned to 35b8b99117)