thanos-io/thanos · warning

Unknown

Unknown

Error message

send series response

What it means

In the store gateway's Series gRPC stream, srv.Send fails while streaming a merged series response chunk to the client; the error is wrapped and converted to a gRPC status with code Unknown. This almost always means the client disconnected or the streaming context was cancelled — the server can no longer deliver the response.

Solutions

1. Treat as benign if logs correlate with query cancellation; verify the client really canceled (check querier logs for context canceled).
2. If unexpected, check network stability between querier and store gateway (LB idle timeouts, proxying of gRPC streams).
3. Increase client-side query/stream timeouts and ensure load balancers support long-lived gRPC streams (disable HTTP/1.1 conversion, raise idle timeout).
4. Reduce stream size (narrow time range/matchers) so clients do not time out mid-stream.

Example fix

// before: client with short timeout cancels stream
timeoutCtx, cancel := context.WithTimeout(ctx, 30*time.Second)
// after: adequate timeout matching expected stream duration
timeoutCtx, cancel := context.WithTimeout(ctx, 5*time.Minute)

if st, ok := status.FromError(err); ok && (st.Code() == codes.Canceled || st.Code() == codes.Unknown) {
    // log at debug: client likely disconnected
}

When it happens

Trigger: StoreAPI Series streaming where the downstream client (querier) cancels its context, closes the connection, or the client is OOM-killed mid-stream; also transport-level failures (broken TCP connection, idle timeout).

Common situations: Users canceling Prometheus/Thanos queries in a UI; querier deadlines cutting off long-running fan-out queries; client crash under memory pressure while receiving large series streams; load balancer dropping long-lived gRPC streams.

Related errors

• Unknown
• sending metadata error to server
• iterate series for block
• %s
• empty exemplars data

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/5ff6f9f444394ca4. Report an issue: GitHub.

Appendix: source

Thrown at pkg/store/bucket.go:1807

			warn := at.GetWarning()
			if warn != "" {
				// TODO(fpetkovski): Consider deprecating string based warnings in favor of a
				// separate protobuf message containing the grpc code and
				// a human readable error message.
				err = status.Error(storepb.GRPCCodeFromWarn(warn), at.GetWarning())
				return
			}

			series := at.GetSeries()
			if series != nil {
				stats.mergedSeriesCount++
				if !req.SkipChunks {
					stats.mergedChunksCount += len(series.Chunks)
					s.metrics.chunkSizeBytes.WithLabelValues(tenant).Observe(float64(chunksSize(series.Chunks)))
				}
			}
			if err = srv.Send(at); err != nil {
				err = status.Error(codes.Unknown, errors.Wrap(err, "send series response").Error())
				return
			}
		}
		stats.MergeDuration = time.Since(begin)
		s.metrics.seriesMergeDuration.WithLabelValues(tenant).Observe(stats.MergeDuration.Seconds())

		err = nil
	})
	if err != nil {
		return err
	}

	if s.enableSeriesResponseHints {
		var anyHints *types.Any

		if queryStatsEnabled {
			resHints.QueryStats = stats.toHints()
		}

View on GitHub (pinned to 35b8b99117)