thanos-io/thanos · error

sending targets error to server

Error message

sending targets error to server %v

What it means

This error is thrown when, after receiving a targets error from the upstream client under the WARNING partial-response strategy, the proxy fails to send the error back to the caller as a warning TargetsResponse. It wraps the Send failure on the downstream gRPC server stream.

Solutions

1. Inspect the wrapped cause for stream cancellation or broken pipe; the original upstream error is likely secondary.
2. Check whether the downstream client (query frontend/UI) is canceling requests and why (timeout, user abort).
3. Ensure the gRPC connection to the client is healthy and no proxies/LBs terminate idle streams.
4. This is mostly unavoidable cleanup noise when clients disconnect; add retry/backoff at the caller instead.

if err := stream.server.Send(resp); err != nil {
    if errors.Is(errors.Cause(err), context.Canceled) {
        return nil // client went away, not actionable
    }
    return err
}

When it happens

Trigger: In receive(), after wrapping a Recv error, PartialResponseStrategy is not ABORT, and stream.server.Send(targetspb.NewWarningTargetsResponse(err)) returns an error (client disconnected, context canceled, broken stream).

Common situations: The querying client canceled the request or timed out while the upstream error was being relayed; gRPC stream broken mid-response; client process restarted.

Related errors

• receiving targets from targets client
• sending targets warning to server
• %s
• empty exemplars data
• Unknown

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/9222987cf2dc0b58. Report an issue: GitHub.

Appendix: source

Thrown at pkg/targets/proxy.go:120

		// Not an error if response strategy is warning.
		return nil
	}

	for {
		target, err := targets.Recv()
		if err == io.EOF {
			return nil
		}

		if err != nil {
			err = errors.Wrapf(err, "receiving targets from targets client %v", stream.client)

			if stream.request.PartialResponseStrategy == storepb.PartialResponseStrategy_ABORT {
				return err
			}

			if err := stream.server.Send(targetspb.NewWarningTargetsResponse(err)); err != nil {
				return errors.Wrapf(err, "sending targets error to server %v", stream.server)
			}
			// Not an error if response strategy is warning.
			return nil
		}

		if w := target.GetWarning(); w != "" {
			if err := stream.server.Send(targetspb.NewWarningTargetsResponse(errors.New(w))); err != nil {
				return errors.Wrapf(err, "sending targets warning to server %v", stream.server)
			}
			continue
		}

		select {
		case stream.channel <- target.GetTargets():
		case <-ctx.Done():
			return ctx.Err()
		}
	}

View on GitHub (pinned to 35b8b99117)