What causes "send series response" in thanos-io/thanos?

Check whether the client canceled — if so this error is benign and can be logged at debug level

send series response

Error message

send series response

What it means

Inside the proxy Store's Series handler, each response received from a downstream store is streamed back to the caller via srv.Send. If Send fails (client disconnected, gRPC stream broken, deadline exceeded), the handler logs and returns an Unknown gRPC status wrapping the error as "send series response". It indicates the stream to the client broke mid-response, not a downstream store problem.

Solutions

Check whether the client canceled — if so this error is benign and can be logged at debug level
Increase client gRPC deadline/timeout to cover full result streaming
Narrow the query (smaller time range, more matchers) to reduce streamed volume
Check network path (LB idle timeout, grpc keepalive settings) for prematurely closed streams

Example fix

// before: short client deadline kills the stream
ctx, cancel := context.WithTimeout(ctx, 5*time.Second)
// after: deadline proportional to expected result size
ctx, cancel := context.WithTimeout(ctx, 120*time.Second)

Defensive patterns

Strategy: try-catch

Try / catch

_, err := queryClient.Series(ctx, req)
if err != nil && strings.Contains(err.Error(), "send series response") {
    // client stream broke: check if ctx was canceled (benign) or network issue (retry with longer deadline)
    if ctx.Err() != nil { log.Debug("query canceled by client") } else { retry with increased deadline }
}

Prevention

Set client gRPC deadlines longer than expected full-response streaming time
Configure gRPC keepalive and LB idle timeouts above query duration
Reduce result volume with tighter matchers/time ranges
Treat cancellation-induced sends as benign and log at debug level

When it happens

Trigger: Client cancels the query or its context expires while responses are being streamed; network interruption between the client and the Query proxy; client-side deadline shorter than the time needed to stream all series.

Common situations: Grafana/UI users canceling large queries; grpc client max response size or deadline too small; load balancer idle timeouts killing long streams; query aborted in ABORT mode causing client to close early.

Related errors

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/0d4993a261c27acb. Report an issue: GitHub.

Appendix: source

Thrown at pkg/store/proxy.go:406

	if s.enableDedup {
		respHeap = NewResponseDeduplicator(respHeap)
	}

	i := 0
	for respHeap.Next() {
		i++
		if r.Limit > 0 && i > int(r.Limit) {
			break
		}
		resp := respHeap.At()

		if resp.GetWarning() != "" && (r.PartialResponseDisabled || r.PartialResponseStrategy == storepb.PartialResponseStrategy_ABORT) {
			return status.Error(codes.Aborted, resp.GetWarning())
		}

		if err := srv.Send(resp); err != nil {
			level.Error(reqLogger).Log("msg", "failed to stream response", "error", err)
			return status.Error(codes.Unknown, errors.Wrap(err, "send series response").Error())
		}
	}

	// Flush any remaining buffered series from the batchable server.
	if f, ok := srv.(flushableServer); ok {
		return f.Flush()
	}

	return nil
}

// LabelNames returns all known label names.
func (s *ProxyStore) LabelNames(ctx context.Context, originalRequest *storepb.LabelNamesRequest) (*storepb.LabelNamesResponse, error) {
	// TODO(bwplotka): This should be part of request logger, otherwise it does not make much sense. Also, could be
	// triggered by tracing span to reduce cognitive load.
	reqLogger := log.With(s.logger, "component", "proxy")
	if s.debugLogging {
		reqLogger = log.With(reqLogger, "request", originalRequest.String())

View on GitHub (pinned to 35b8b99117)