thanos-io/thanos · error

sending rules error to server

Error message

sending rules error to server %v

What it means

Raised in rulesStream.receive after an upstream Recv error when the strategy is not ABORT: the code forwards the failure as a WarningRulesResponse via stream.server.Send, and that send to the client itself fails. This masks the original receive error with a second transport failure.

Solutions

1. Check why the downstream gRPC server stream is broken (client gone, context canceled)
2. Log both the original upstream error and the send failure for diagnosis
3. Consider buffering/limiting warnings if slow clients cause send errors

When it happens

Trigger: Thrown at pkg/rules/proxy.go:136 when the library encounters an invalid state.

Common situations: See trigger scenarios.

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/2f247893ba309611. Report an issue: GitHub.

Appendix: source

Thrown at pkg/rules/proxy.go:136

	for {
		rule, err := rules.Recv()
		if err == io.EOF {
			return nil
		}

		if err != nil {
			// An error happened in Recv(), hence the underlying stream is aborted
			// as per https://github.com/grpc/grpc-go/blob/7f2581f910fc21497091c4109b56d310276fc943/stream.go#L117-L125.
			// We must not continue receiving additional data from it and must return.
			err = errors.Wrapf(err, "receiving rules from rules client %v", stream.client)

			if stream.request.PartialResponseStrategy == storepb.PartialResponseStrategy_ABORT {
				return err
			}

			if err := stream.server.Send(rulespb.NewWarningRulesResponse(err)); err != nil {
				return errors.Wrapf(err, "sending rules error to server %v", stream.server)
			}

			// Return no error if response strategy is warning.
			return nil
		}

		if w := rule.GetWarning(); w != "" {
			if err := stream.server.Send(rulespb.NewWarningRulesResponse(errors.New(w))); err != nil {
				return errors.Wrapf(err, "sending rules warning to server %v", stream.server)
			}
			// Client stream is not aborted, it is ok to receive additional data.
			continue
		}

		select {
		case stream.channel <- rule.GetGroup():
		case <-ctx.Done():
			return ctx.Err()

View on GitHub (pinned to 35b8b99117)