{"record":{"id":"55b9ffad7c565a59","repo":"thanos-io/thanos","slug":"sending-rules-warning-to-server-v","errorCode":null,"errorMessage":"sending rules warning to server %v","messagePattern":"sending rules warning to server (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"pkg/rules/proxy.go","lineNumber":145,"sourceCode":"\t\t\t// as per https://github.com/grpc/grpc-go/blob/7f2581f910fc21497091c4109b56d310276fc943/stream.go#L117-L125.\n\t\t\t// We must not continue receiving additional data from it and must return.\n\t\t\terr = errors.Wrapf(err, \"receiving rules from rules client %v\", stream.client)\n\n\t\t\tif stream.request.PartialResponseStrategy == storepb.PartialResponseStrategy_ABORT {\n\t\t\t\treturn err\n\t\t\t}\n\n\t\t\tif err := stream.server.Send(rulespb.NewWarningRulesResponse(err)); err != nil {\n\t\t\t\treturn errors.Wrapf(err, \"sending rules error to server %v\", stream.server)\n\t\t\t}\n\n\t\t\t// Return no error if response strategy is warning.\n\t\t\treturn nil\n\t\t}\n\n\t\tif w := rule.GetWarning(); w != \"\" {\n\t\t\tif err := stream.server.Send(rulespb.NewWarningRulesResponse(errors.New(w))); err != nil {\n\t\t\t\treturn errors.Wrapf(err, \"sending rules warning to server %v\", stream.server)\n\t\t\t}\n\t\t\t// Client stream is not aborted, it is ok to receive additional data.\n\t\t\tcontinue\n\t\t}\n\n\t\tselect {\n\t\tcase stream.channel <- rule.GetGroup():\n\t\tcase <-ctx.Done():\n\t\t\treturn ctx.Err()\n\t\t}\n\t}\n}\n","sourceCodeStart":127,"sourceCodeEnd":158,"githubUrl":"https://github.com/thanos-io/thanos/blob/35b8b991177def87ed52dcf10f9b6d87f07282c8/pkg/rules/proxy.go#L127-L158","documentation":"When a rules client stream receives a rule group carrying a warning (rule.GetWarning() != \"\"), the proxy forwards a NewWarningRulesResponse back to the server and wraps any Send failure with this message. It means the gRPC stream to the server broke while delivering a warning response; the receive loop itself continues so additional data can still be processed.","triggerScenarios":"stream.recv delivers a group whose Warning field is set, and stream.server.Send(rulespb.NewWarningRulesResponse(...)) fails (stream torn down, client disconnected, connection reset).","commonSituations":"Upstream ruler sends a warning (e.g. truncated rule response) while the gRPC connection is flaky or already closing; client cancelled the RPC mid-stream; network interruption between proxy and server.","solutions":["Check connectivity/keepalive settings between the rules client and server; retry the Rules request","Inspect server logs for the wrapped underlying gRPC error to identify the transport cause","If the stream was cancelled by the caller, treat this as an expected cancellation rather than a bug"],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"if streamClosed { return status.Error(codes.Aborted, \"stream already closed\") }","typeGuard":null,"tryCatchPattern":"if err := doRulesRequest(); err != nil {\n    if errors.Is(err, context.Canceled) || status.Code(err) == codes.Canceled {\n        return nil // expected cancellation\n    }\n    var wrapped interface{ Unwrap() error }\n    log.Printf(\"warning send failed: %v\", err)\n    return retryRulesRequest()\n}","preventionTips":["Configure gRPC keepalive to detect dead streams early","Treat cancellations separately from real transport failures","Monitor reconnection/stream-error metrics on the rules client"],"tags":["grpc","network","streaming"],"backgroundTag":"broken-pipe","analyzedSha":"35b8b991177def87ed52dcf10f9b6d87f07282c8","analyzedAt":"2026-09-07T01:49:59.689Z","contentChangedAt":"2026-09-07T01:49:59.689Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}