{"record":{"id":"5ff6f9f444394ca4","repo":"thanos-io/thanos","slug":"unknown-5ff6f9","errorCode":"Unknown","errorMessage":"send series response","messagePattern":"send series response","errorType":"http","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"pkg/store/bucket.go","lineNumber":1807,"sourceCode":"\t\t\twarn := at.GetWarning()\n\t\t\tif warn != \"\" {\n\t\t\t\t// TODO(fpetkovski): Consider deprecating string based warnings in favor of a\n\t\t\t\t// separate protobuf message containing the grpc code and\n\t\t\t\t// a human readable error message.\n\t\t\t\terr = status.Error(storepb.GRPCCodeFromWarn(warn), at.GetWarning())\n\t\t\t\treturn\n\t\t\t}\n\n\t\t\tseries := at.GetSeries()\n\t\t\tif series != nil {\n\t\t\t\tstats.mergedSeriesCount++\n\t\t\t\tif !req.SkipChunks {\n\t\t\t\t\tstats.mergedChunksCount += len(series.Chunks)\n\t\t\t\t\ts.metrics.chunkSizeBytes.WithLabelValues(tenant).Observe(float64(chunksSize(series.Chunks)))\n\t\t\t\t}\n\t\t\t}\n\t\t\tif err = srv.Send(at); err != nil {\n\t\t\t\terr = status.Error(codes.Unknown, errors.Wrap(err, \"send series response\").Error())\n\t\t\t\treturn\n\t\t\t}\n\t\t}\n\t\tstats.MergeDuration = time.Since(begin)\n\t\ts.metrics.seriesMergeDuration.WithLabelValues(tenant).Observe(stats.MergeDuration.Seconds())\n\n\t\terr = nil\n\t})\n\tif err != nil {\n\t\treturn err\n\t}\n\n\tif s.enableSeriesResponseHints {\n\t\tvar anyHints *types.Any\n\n\t\tif queryStatsEnabled {\n\t\t\tresHints.QueryStats = stats.toHints()\n\t\t}","sourceCodeStart":1789,"sourceCodeEnd":1825,"githubUrl":"https://github.com/thanos-io/thanos/blob/35b8b991177def87ed52dcf10f9b6d87f07282c8/pkg/store/bucket.go#L1789-L1825","documentation":"In the store gateway's Series gRPC stream, srv.Send fails while streaming a merged series response chunk to the client; the error is wrapped and converted to a gRPC status with code Unknown. This almost always means the client disconnected or the streaming context was cancelled — the server can no longer deliver the response.","triggerScenarios":"StoreAPI Series streaming where the downstream client (querier) cancels its context, closes the connection, or the client is OOM-killed mid-stream; also transport-level failures (broken TCP connection, idle timeout).","commonSituations":"Users canceling Prometheus/Thanos queries in a UI; querier deadlines cutting off long-running fan-out queries; client crash under memory pressure while receiving large series streams; load balancer dropping long-lived gRPC streams.","solutions":["Treat as benign if logs correlate with query cancellation; verify the client really canceled (check querier logs for context canceled).","If unexpected, check network stability between querier and store gateway (LB idle timeouts, proxying of gRPC streams).","Increase client-side query/stream timeouts and ensure load balancers support long-lived gRPC streams (disable HTTP/1.1 conversion, raise idle timeout).","Reduce stream size (narrow time range/matchers) so clients do not time out mid-stream."],"exampleFix":"// before: client with short timeout cancels stream\ntimeoutCtx, cancel := context.WithTimeout(ctx, 30*time.Second)\n// after: adequate timeout matching expected stream duration\ntimeoutCtx, cancel := context.WithTimeout(ctx, 5*time.Minute)","handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"if st, ok := status.FromError(err); ok && (st.Code() == codes.Canceled || st.Code() == codes.Unknown) {\n    // log at debug: client likely disconnected\n}","preventionTips":["Configure load balancers for long-lived gRPC streams (high idle timeout)","Use consistent timeouts across querier and store","Monitor client cancellation rates to distinguish benign vs real failures"],"tags":["grpc","streaming","thanos-store","client-disconnect"],"backgroundTag":"broken-pipe","analyzedSha":"35b8b991177def87ed52dcf10f9b6d87f07282c8","analyzedAt":"2026-09-07T01:49:59.689Z","contentChangedAt":"2026-09-07T01:49:59.689Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}