thanos-io/thanos · error

sending tsdb statistics error to server

Error message

sending tsdb statistics error to server %v

What it means

When an upstream stream error occurs under the non-ABORT (warning) strategy, the proxy forwards it to the caller as a warning response. If that Send itself fails, the error is wrapped as 'sending tsdb statistics error to server %v'. This is a secondary failure — the proxy could not even deliver the warning.

Solutions

1. Investigate why the downstream client connection terminated (client timeout, cancel, crash).
2. Check proxy-to-client network path and idle timeout settings.
3. Ensure clients keep the stream open until receiving all responses or explicitly cancel.
4. Treat as benign if the client merely gave up early; fix the original upstream error at the store.

Example fix

// before
ctx, cancel := context.WithTimeout(ctx, 2*time.Second) // client gives up before warnings flush
// after
ctx, cancel := context.WithTimeout(ctx, 30*time.Second) // allow full stream + warnings to be delivered

if err := call(ctx); err != nil {
    if strings.Contains(err.Error(), "sending tsdb statistics error to server") {
        // downstream client already gone; log and stop
        logger.Debug("client disconnected during stats proxying", "err", err)
        return nil
    }
    return err
}

When it happens

Trigger: In receive(), after wrapping an upstream statsClient error and strategy is WARNINGS, `stream.server.Send(NewWarningTSDBStatisticsResponse(err))` fails because the downstream client disconnected or its stream is broken.

Common situations: Client canceled the request while the proxy was handling a store failure; downstream gRPC transport died concurrently; load balancer closed the client connection.

Related errors

• receiving tsdb statistics from status client
• sending tsdb statistics warning to server
• Unknown
• %s
• empty exemplars data

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/8628e7185ca0904d. Report an issue: GitHub.

Appendix: source

Thrown at pkg/status/proxy.go:130

		// Not an error if response strategy is warning.
		return nil
	}

	for {
		resp, err := statsClient.Recv()
		if err == io.EOF {
			return nil
		}

		if err != nil {
			err = errors.Wrapf(err, "receiving tsdb statistics from status client %v", stream.client)

			if stream.request.PartialResponseStrategy == storepb.PartialResponseStrategy_ABORT {
				return err
			}

			if err := stream.server.Send(statuspb.NewWarningTSDBStatisticsResponse(err)); err != nil {
				return errors.Wrapf(err, "sending tsdb statistics error to server %v", stream.server)
			}

			// Not an error if response strategy is warning.
			return nil
		}

		if w := resp.GetWarning(); w != "" {
			if err := stream.server.Send(statuspb.NewWarningTSDBStatisticsResponse(errors.New(w))); err != nil {
				return errors.Wrapf(err, "sending tsdb statistics warning to server %v", stream.server)
			}
			continue
		}

		select {
		case stream.channel <- resp.GetStatistics():
		case <-ctx.Done():
			return ctx.Err()
		}

View on GitHub (pinned to 35b8b99117)