What causes "send targets response" in thanos-io/thanos?

Check whether the client canceled (context.Canceled) — that is benign and can be ignored/logged at debug level

Unknown

Unknown

Error message

send targets response

What it means

The targets proxy's Targets gRPC handler streams each fetched target to the client via srv.Send. If any send fails, it wraps the error as 'send targets response' with a gRPC Unknown status. The failure is on the server->client stream leg, e.g. the client disconnected.

Solutions

Check whether the client canceled (context.Canceled) — that is benign and can be ignored/logged at debug level
Verify network connectivity and keepalive settings between client and proxy
Reduce response size (fewer targets, pagination) to avoid long-lived streams
Update client/server grpc-go versions to fix stream teardown bugs

Defensive patterns

Strategy: retry

Try / catch

if err := srv.Send(resp); err != nil { if status.Code(err) == codes.Canceled || errors.Is(err, context.Canceled) { return nil } return status.Error(codes.Unknown, errors.Wrap(err, "send targets response").Error()) }

Prevention

Treat client cancellation as normal, not an alert-worthy failure
Tune gRPC keepalive to survive idle streams
Limit target set size per stream

When it happens

Trigger: Calling Targets and having srv.Send return a non-nil error — typically the client canceled the context/closed the connection mid-stream, or the transport broke while streaming targets.

Common situations: Query cancellations in Grafana or the querier tearing down gRPC streams, network partitions between querier and targets proxy, deadlines exceeded during large target sets.

Related errors

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/47299ea8498f3252. Report an issue: GitHub.

Appendix: source

Thrown at pkg/targets/proxy.go:76

	go func() {
		_ = g.Wait()
		close(respChan)
	}()

	for resp := range respChan {
		// TODO: Stream it
		targets = append(targets, resp)
	}

	if err := g.Wait(); err != nil {
		level.Error(s.logger).Log("err", err)
		return err
	}

	for _, t := range targets {
		if err := srv.Send(targetspb.NewTargetsResponse(t)); err != nil {
			return status.Error(codes.Unknown, errors.Wrap(err, "send targets response").Error())
		}
	}

	return nil
}

type targetsStream struct {
	client  targetspb.TargetsClient
	request *targetspb.TargetsRequest
	channel chan<- *targetspb.TargetDiscovery
	server  targetspb.Targets_TargetsServer
}

func (stream *targetsStream) receive(ctx context.Context) error {
	targets, err := stream.client.Targets(ctx, stream.request)
	if err != nil {
		err = errors.Wrapf(err, "fetching targets from targets client %v", stream.client)

View on GitHub (pinned to 35b8b99117)