thanos-io/thanos · error

proxy Targets

Error message

proxy Targets

What it means

This error wraps any failure produced by rr.proxy.Targets() during a proxied TargetDiscovery request through a Rules/targets replica-aware proxy. It means the fan-out to underlying TargetsAPI clients (gathering active/dropped targets) failed, e.g. no clients or all replicas returned errors.

Solutions

1. Check the wrapped cause: if 'no such subscription' or 'no endpoints', no targets providers are registered — verify store/sidecar discovery.
2. Ensure sidecars/store gateways are up and registered via the same mDNS/DNS service discovery as the querier.
3. Use PartialResponseStrategy_WARNING to tolerate some endpoint failures.
4. Check network connectivity and gRPC health of all listed endpoints (thanos tools bucket verify / query UI store page).

Example fix

// before
client, err := proxy.Targets(ctx, req)
// after: tolerate partial failures
req.PartialResponseStrategy = storepb.PartialResponseStrategy_WARNING
client, err := proxy.Targets(ctx, req)

// Verify at least one targets provider is registered before fan-out
if len(endpoints) == 0 { return nil, errors.New("no targets endpoints registered") }

resp, err := proxy.Targets(ctx, req)
if err != nil {
    log.Warn("targets proxy failed, returning empty", zap.Error(err))
    return &targetspb.TargetDiscovery{}, nil
}

When it happens

Trigger: GRPCProxy.Targets (via TargetsServer in pkg/targets/targets.go:57) returns a non-nil error from proxying the Targets request to backend store/targets endpoints.

Common situations: No healthy TargetsAPI endpoints registered (all sidecars down); all replicas failed so proxy returns error; context deadline during fan-out; querier cannot discover store endpoints via mDNS/DNS.

Related errors

• proxy Exemplars
• Aborted
• proxy MetricMetadata
• fetching metric metadata from metadata client
• proxy Rules

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/bfb3062f50806176. Report an issue: GitHub.

Appendix: source

Thrown at pkg/targets/targets.go:57

	c := &GRPCClient{
		proxy:         ts,
		replicaLabels: map[string]struct{}{},
	}

	for _, label := range replicaLabels {
		c.replicaLabels[label] = struct{}{}
	}
	return c
}

func (rr *GRPCClient) Targets(ctx context.Context, req *targetspb.TargetsRequest) (*targetspb.TargetDiscovery, annotations.Annotations, error) {
	resp := &targetsServer{ctx: ctx, targets: &targetspb.TargetDiscovery{
		ActiveTargets:  make([]*targetspb.ActiveTarget, 0),
		DroppedTargets: make([]*targetspb.DroppedTarget, 0),
	}}

	if err := rr.proxy.Targets(req, resp); err != nil {
		return nil, nil, errors.Wrap(err, "proxy Targets")
	}

	resp.targets = dedupTargets(resp.targets, rr.replicaLabels)

	return resp.targets, resp.warnings, nil
}

// dedupTargets re-sorts the set so that the same target with different replica
// labels are coming right after each other.
func dedupTargets(targets *targetspb.TargetDiscovery, replicaLabels map[string]struct{}) *targetspb.TargetDiscovery {
	if targets == nil {
		return nil
	}

	targets.ActiveTargets = dedupActiveTargets(targets.ActiveTargets, replicaLabels)
	targets.DroppedTargets = dedupDroppedTargets(targets.DroppedTargets, replicaLabels)

	return targets

View on GitHub (pinned to 35b8b99117)