thanos-io/thanos · error

receiving targets from targets client

Error message

receiving targets from targets client %v

What it means

This error wraps a failure from the gRPC Recv() call while a Thanos proxy StreamsTargets from an upstream targets client (e.g. a sidecar or store). The underlying error may be a transport failure, context cancellation, or the upstream server closing with an error. Depending on PartialResponseStrategy it either aborts the stream or is downgraded to a warning response.

Solutions

1. Check the wrapped cause error to identify the upstream failure (connectivity, deadline, cancellation).
2. Verify the upstream TargetsAPI (sidecar/store) address in the store/sidecar config is reachable: curl/tonic the gRPC endpoint and check --grpc-address flags.
3. Check query timeout settings (--query.timeout) and increase if large target lists cause deadline exceeded.
4. If failures should not abort the whole response, use PartialResponseStrategy_WARNING so the error is sent as a warning instead.
5. Inspect gRPC logs and TLS certificates if the stream fails to authenticate to the upstream.

Example fix

// before: errors.Wrapf(err, "receiving targets from targets client %v", stream.client)
// after (diagnose cause first):
if errors.Is(errors.Cause(err), context.DeadlineExceeded) {
    log.Warnf("targets stream timeout, increasing query timeout may help: %v", err)
}
return err

// Go: check reachability before querying targets
if err := probeGRPC(ctx, sidecarAddr); err != nil { return fmt.Errorf("targets upstream %s unreachable: %w", sidecarAddr, err) }

if err != nil {
    if errors.Is(errors.Cause(err), context.DeadlineExceeded) || errors.Is(errors.Cause(err), context.Canceled) {
        return nil, fmt.Errorf("targets stream canceled: %w", err)
    }
    return nil, err
}

When it happens

Trigger: TargetsServer.receive() receives a non-EOF error from targets.Recv() on the gRPC stream, e.g. upstream unreachable, stream reset, or context deadline exceeded.

Common situations: The queried TargetsAPI endpoint (sidecar/store gateway) is down or restarting; network interruption between query and targets source; gRPC context canceled due to query timeout; TLS or auth misconfiguration to the upstream.

Related errors

• receiving metric metadata from metadata client
• sending rules warning to server
• Unknown
• receiving tsdb statistics from status client
• iterate series for block

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/ab14c89dcebaa68a. Report an issue: GitHub.

Appendix: source

Thrown at pkg/targets/proxy.go:113

		if stream.request.PartialResponseStrategy == storepb.PartialResponseStrategy_ABORT {
			return err
		}

		if serr := stream.server.Send(targetspb.NewWarningTargetsResponse(err)); serr != nil {
			return serr
		}
		// Not an error if response strategy is warning.
		return nil
	}

	for {
		target, err := targets.Recv()
		if err == io.EOF {
			return nil
		}

		if err != nil {
			err = errors.Wrapf(err, "receiving targets from targets client %v", stream.client)

			if stream.request.PartialResponseStrategy == storepb.PartialResponseStrategy_ABORT {
				return err
			}

			if err := stream.server.Send(targetspb.NewWarningTargetsResponse(err)); err != nil {
				return errors.Wrapf(err, "sending targets error to server %v", stream.server)
			}
			// Not an error if response strategy is warning.
			return nil
		}

		if w := target.GetWarning(); w != "" {
			if err := stream.server.Send(targetspb.NewWarningTargetsResponse(errors.New(w))); err != nil {
				return errors.Wrapf(err, "sending targets warning to server %v", stream.server)
			}
			continue
		}

View on GitHub (pinned to 35b8b99117)