{"record":{"id":"3eca69fc407a1f8a","repo":"thanos-io/thanos","slug":"next-proto","errorCode":null,"errorMessage":"next proto","messagePattern":"next proto","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"pkg/store/prometheus.go","lineNumber":313,"sourceCode":"\n\tvar data = p.getBuffer()\n\tdefer p.putBuffer(data)\n\n\tbodySizer := NewBytesRead(httpResp.Body)\n\tseriesStats := &storepb.SeriesStatsCounter{}\n\n\t// TODO(bwplotka): Put read limit as a flag.\n\tstream := remote.NewChunkedReader(bodySizer, config.DefaultChunkedReadLimit, *data)\n\thasher := hashPool.Get().(hash.Hash64)\n\tdefer hashPool.Put(hasher)\n\tfor {\n\t\tres := &prompb.ChunkedReadResponse{}\n\t\terr := stream.NextProto(res)\n\t\tif err == io.EOF {\n\t\t\tbreak\n\t\t}\n\t\tif err != nil {\n\t\t\treturn errors.Wrap(err, \"next proto\")\n\t\t}\n\n\t\tif len(res.ChunkedSeries) != 1 {\n\t\t\tlevel.Warn(p.logger).Log(\"msg\", \"Prometheus ReadRequest_STREAMED_XOR_CHUNKS returned non 1 series in frame\", \"series\", len(res.ChunkedSeries))\n\t\t}\n\n\t\tframesNum++\n\t\tfor _, series := range res.ChunkedSeries {\n\t\t\t// MergeLabels() prefers local labels over external labels but we prefer\n\t\t\t// external labels hence we need to do this:\n\t\t\t// https://github.com/prometheus/prometheus/blob/3f6f5d3357e232abe53f1775f893fdf8f842712c/storage/remote/codec.go#L210.\n\t\t\tcompleteLabelset := rmLabels(labelpb.ExtendSortedLabels(labelpb.ZLabelsToPromLabels(series.Labels), extLset), extLsetToRemove)\n\t\t\tif !shardMatcher.MatchesLabels(completeLabelset) {\n\t\t\t\tcontinue\n\t\t\t}\n\n\t\t\tseriesStats.CountSeries(series.Labels)\n\t\t\tthanosChks := make([]storepb.AggrChunk, len(series.Chunks))","sourceCodeStart":295,"sourceCodeEnd":331,"githubUrl":"https://github.com/thanos-io/thanos/blob/35b8b991177def87ed52dcf10f9b6d87f07282c8/pkg/store/prometheus.go#L295-L331","documentation":"handleStreamedPrometheusResponse iterates a streamed chunked remote-read response calling stream.NextProto for each frame. Any decode/transport error other than io.EOF is wrapped as 'next proto', meaning a frame of the ChunkedReadResponse stream failed to be read or unmarshaled.","triggerScenarios":"During a Series call with streamed chunked remote read: connection reset mid-stream, truncated/garbled frame, server closing the stream unexpectedly, or a payload failing protobuf decode into prompb.ChunkedReadResponse.","commonSituations":"Load balancers/idle-timeout killing long streaming responses; Prometheus restarting or OOMing mid-query; proxies buffering/breaking chunked transfer; network instability on large queries.","solutions":["Check the wrapped cause (%w) — connection reset vs unmarshal error point to network vs data issues.","Inspect Prometheus server logs for restarts/errors during the query window.","Reduce query time range/scope to shrink the streamed response, or shard the query.","Check intermediary proxies/LBs for idle or response-size timeouts on streaming connections and raise them.","Retry the query; transient network resets are common on long streams."],"exampleFix":null,"handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"if err := s.Series(ctx, req); err != nil {\n    if strings.Contains(err.Error(), \"next proto\") {\n        // transient stream break: retry with backoff; narrow the time range if persistent\n    }\n    return err\n}","preventionTips":["Raise LB/proxy idle timeouts for long streaming gRPC-over-HTTP responses.","Keep query time ranges bounded to reduce stream duration.","Monitor Prometheus restarts/OOMs during heavy queries.","Use retry with backoff for transient stream failures."],"tags":["prometheus","remote-read","streaming","protobuf","network"],"backgroundTag":"stream-read-failed","analyzedSha":"35b8b991177def87ed52dcf10f9b6d87f07282c8","analyzedAt":"2026-09-07T01:49:59.689Z","contentChangedAt":"2026-09-07T01:49:59.689Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}