thanos-io/thanos · error

Unknown

Unknown

Error message

send tsdb statistics response

What it means

TSDBStatistics (gRPC server side of the status proxy) streams per-store statistics responses; if srv.Send fails while forwarding a stat, the error is wrapped as 'send tsdb statistics response' and returned with gRPC code Unknown. It means the server could not deliver a statistics message on the stream.

Solutions

1. Check client-side logs for the root cause (context deadline exceeded, connection reset) — this error is a symptom of the client/transport failure.
2. Increase the client request timeout for large TSDB statistics requests.
3. Verify network connectivity and load balancer idle timeouts between client and server.
4. Retry the TSDB statistics request; the stream is safe to re-issue.

Example fix

// before
ctx := context.Background()
resp, err := client.TSDBStatistics(ctx, req)
// after
ctx, cancel := context.WithTimeout(ctx, 60*time.Second)
defer cancel()
resp, err := client.TSDBStatistics(ctx, req)

var lastErr error
for i := 0; i < 3; i++ {
    ctx, cancel := context.WithTimeout(ctx, 30*time.Second)
    err := fetchTSDBStats(ctx)
    cancel()
    if err == nil || !isTransportErr(err) { break }
    lastErr = err
    time.Sleep(backoff(i))
}

When it happens

Trigger: Client disconnects or cancels the context mid-stream; gRPC transport error (broken connection, deadline exceeded) while the server loops `srv.Send(statuspb.NewTSDBStatisticsResponse(stat))` for each stat.

Common situations: Client timeout too short for large clusters with many stores; network interruption between client and querier; client process crash during TSDB status fetch; context cancellation in a UI request that timed out.

Related errors

• receiving metric metadata from metadata client
• sending rules warning to server
• receiving tsdb statistics from status client
• no tsdb statistics
• iterate series for block

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/9a026da0ec26bbb6. Report an issue: GitHub.

Appendix: source

Thrown at pkg/status/proxy.go:77

		_ = g.Wait()
		close(respChan)
	}()

	for resp := range respChan {
		statistics = append(statistics, resp)
	}

	if err := g.Wait(); err != nil {
		level.Error(s.logger).Log("err", err)
		return err
	}

	for _, stat := range statistics {
		tracing.DoInSpan(srv.Context(), "send_tsdb_statistics_response", func(_ context.Context) {
			err = srv.Send(statuspb.NewTSDBStatisticsResponse(stat))
		})
		if err != nil {
			return status.Error(codes.Unknown, errors.Wrap(err, "send tsdb statistics response").Error())
		}
	}

	return nil
}

type statsStream struct {
	client  statuspb.StatusClient
	request *statuspb.TSDBStatisticsRequest
	channel chan<- *statuspb.TSDBStatistics
	server  statuspb.Status_TSDBStatisticsServer
}

func (stream *statsStream) receive(ctx context.Context) error {
	var (
		err         error
		statsClient statuspb.Status_TSDBStatisticsClient
	)

View on GitHub (pinned to 35b8b99117)