thanos-io/thanos · warning

sending rules warning to server

Error message

sending rules warning to server %v

What it means

When a rules client stream receives a rule group carrying a warning (rule.GetWarning() != ""), the proxy forwards a NewWarningRulesResponse back to the server and wraps any Send failure with this message. It means the gRPC stream to the server broke while delivering a warning response; the receive loop itself continues so additional data can still be processed.

Solutions

1. Check connectivity/keepalive settings between the rules client and server; retry the Rules request
2. Inspect server logs for the wrapped underlying gRPC error to identify the transport cause
3. If the stream was cancelled by the caller, treat this as an expected cancellation rather than a bug

if streamClosed { return status.Error(codes.Aborted, "stream already closed") }

if err := doRulesRequest(); err != nil {
    if errors.Is(err, context.Canceled) || status.Code(err) == codes.Canceled {
        return nil // expected cancellation
    }
    var wrapped interface{ Unwrap() error }
    log.Printf("warning send failed: %v", err)
    return retryRulesRequest()
}

When it happens

Trigger: stream.recv delivers a group whose Warning field is set, and stream.server.Send(rulespb.NewWarningRulesResponse(...)) fails (stream torn down, client disconnected, connection reset).

Common situations: Upstream ruler sends a warning (e.g. truncated rule response) while the gRPC connection is flaky or already closing; client cancelled the RPC mid-stream; network interruption between proxy and server.

Related errors

• receiving metric metadata from metadata client
• Unknown
• receiving tsdb statistics from status client
• iterate series for block
• send series response

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/55b9ffad7c565a59. Report an issue: GitHub.

Appendix: source

Thrown at pkg/rules/proxy.go:145

			// as per https://github.com/grpc/grpc-go/blob/7f2581f910fc21497091c4109b56d310276fc943/stream.go#L117-L125.
			// We must not continue receiving additional data from it and must return.
			err = errors.Wrapf(err, "receiving rules from rules client %v", stream.client)

			if stream.request.PartialResponseStrategy == storepb.PartialResponseStrategy_ABORT {
				return err
			}

			if err := stream.server.Send(rulespb.NewWarningRulesResponse(err)); err != nil {
				return errors.Wrapf(err, "sending rules error to server %v", stream.server)
			}

			// Return no error if response strategy is warning.
			return nil
		}

		if w := rule.GetWarning(); w != "" {
			if err := stream.server.Send(rulespb.NewWarningRulesResponse(errors.New(w))); err != nil {
				return errors.Wrapf(err, "sending rules warning to server %v", stream.server)
			}
			// Client stream is not aborted, it is ok to receive additional data.
			continue
		}

		select {
		case stream.channel <- rule.GetGroup():
		case <-ctx.Done():
			return ctx.Err()
		}
	}
}

View on GitHub (pinned to 35b8b99117)