VictoriaMetrics/VictoriaMetrics · error

cannot discover Kuma targets

Error message

cannot discover Kuma targets: %w

What it means

During Kuma SD initialization, targets are fetched synchronously once so the first scrape has data. If updateTargetsLabels(ctx) fails (network error, non-200 from the control plane, bad xDS/mesh response), initialization aborts, the client is stopped, and the error is wrapped with this message. It means the initial Kuma target discovery failed.

Solutions

1. Verify the Kuma control plane address/port is reachable from vmagent (curl http://kuma-cp:5676) and fix the `server` value
2. Check the wrapped error to distinguish connection refused/DNS/TLS from bad response data
3. Ensure startup ordering (vmagent starts after the control plane) or retry deployment
4. Inspect control-plane TLS certs and vmagent's CA configuration if handshake errors appear

Example fix

# before: vmagent starts before kuma-cp exists
server: "http://kuma-control-plane:5676"
# after: add readiness gating / correct address
server: "http://kuma-control-plane.kuma-system.svc:5676"

addr := strings.TrimPrefix(strings.TrimPrefix(server, "http://"), "https://")
conn, err := net.DialTimeout("tcp", addr, 3*time.Second)
if err != nil {
    return fmt.Errorf("kuma control plane %s not reachable: %w", addr, err)
}
conn.Close()

for attempt := 0; attempt < 5; attempt++ {
    ac, err := kuma.NewSDConfig(sdc)
    if err == nil {
        return ac, nil
    }
    if strings.Contains(err.Error(), "cannot discover Kuma targets") {
        time.Sleep(time.Duration(attempt+1) * 2 * time.Second)
        continue
    }
    return nil, err
}
return nil, errors.New("kuma control plane unreachable after retries")

When it happens

Trigger: Kuma control plane unreachable (DNS failure, connection refused, TLS handshake error) or returning invalid data when vmagent starts with a kuma_sd_config.

Common situations: Control plane not yet up during vmagent startup (ordering issue in k8s); wrong port; network policy/firewall blocking; expired control-plane mTLS certificates.

Related errors

• cannot get API config for kuma_sd
• error when reading Kuma discovery response
• cannot create HTTP client for
• cannot get clouds
• cannot get organizations

AI-assisted analysis of VictoriaMetrics/VictoriaMetrics@5079fb58f1 (2026-09-03). Data as JSON: /api/errors/b8f9bee5a379a1c3. Report an issue: GitHub.

Appendix: source

Thrown at lib/promscrape/discovery/kuma/api.go:95

	cfg := &apiConfig{
		client:   client,
		clientID: clientID,
		apiPath:  apiPath,

		fetchErrors: metrics.GetOrCreateCounter(fmt.Sprintf(`promscrape_discovery_kuma_errors_total{type="fetch",url=%q}`, sdc.Server)),
		parseErrors: metrics.GetOrCreateCounter(fmt.Sprintf(`promscrape_discovery_kuma_errors_total{type="parse",url=%q}`, sdc.Server)),
	}

	ctx, cancel := context.WithCancel(context.Background())
	cfg.cancel = cancel

	// Initialize targets synchronously and then start updating them in background.
	// The synchronous targets' update is needed for returning non-empty list of targets
	// just after the initialization.
	if err := cfg.updateTargetsLabels(ctx); err != nil {
		client.Stop()
		return nil, fmt.Errorf("cannot discover Kuma targets: %w", err)
	}
	cfg.wg.Go(func() {
		cfg.runTargetsWatcher(ctx)
	})

	return cfg, nil
}

func getAPIServerPath(serverURL string) (string, string, error) {
	if serverURL == "" {
		return "", "", fmt.Errorf("missing server url")
	}
	if !strings.Contains(serverURL, "://") {
		serverURL = "http://" + serverURL
	}
	psu, err := url.Parse(serverURL)
	if err != nil {
		return "", "", fmt.Errorf("cannot parse server url=%q: %w", serverURL, err)

View on GitHub (pinned to 5079fb58f1)