thanos-io/thanos · warning

sending targets warning to server

Error message

sending targets warning to server %v

What it means

This error is thrown when the proxy tries to relay a warning received from an upstream target response to the downstream gRPC server, and that Send call itself fails. The upstream sent a Target with a Warning field; relaying it as a WarningTargetsResponse failed.

Solutions

1. Check the wrapped cause; typically context canceled — trace which caller dropped the stream.
2. Increase query/client timeouts so the stream stays open until warnings are delivered.
3. Verify network paths (ingress gateways) are not closing gRPC streams prematurely; raise idle timeouts.
4. Treat as noise during client shutdown; filter by errors.Is(err, context.Canceled) in logs.

if err := stream.server.Send(warningResp); err != nil {
    log.Warn("failed to relay targets warning", zap.Error(err))
    return nil // do not mask original warning
}

When it happens

Trigger: In receive(), target.GetWarning() is non-empty and stream.server.Send(targetspb.NewWarningTargetsResponse(...)) fails due to a broken/canceled downstream stream.

Common situations: Query client disconnected or timed out exactly when a warning (e.g. partial results from one sidecar) was being forwarded; LB idle timeout killing the stream.

Related errors

• receiving targets from targets client
• sending targets error to server
• %s
• empty exemplars data
• Unknown

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/5af45ea27456cfca. Report an issue: GitHub.

Appendix: source

Thrown at pkg/targets/proxy.go:128

		}

		if err != nil {
			err = errors.Wrapf(err, "receiving targets from targets client %v", stream.client)

			if stream.request.PartialResponseStrategy == storepb.PartialResponseStrategy_ABORT {
				return err
			}

			if err := stream.server.Send(targetspb.NewWarningTargetsResponse(err)); err != nil {
				return errors.Wrapf(err, "sending targets error to server %v", stream.server)
			}
			// Not an error if response strategy is warning.
			return nil
		}

		if w := target.GetWarning(); w != "" {
			if err := stream.server.Send(targetspb.NewWarningTargetsResponse(errors.New(w))); err != nil {
				return errors.Wrapf(err, "sending targets warning to server %v", stream.server)
			}
			continue
		}

		select {
		case stream.channel <- target.GetTargets():
		case <-ctx.Done():
			return ctx.Err()
		}
	}
}

View on GitHub (pinned to 35b8b99117)