VictoriaMetrics/VictoriaMetrics · error

cannot discover Kuma targets: %w

Error message

cannot discover Kuma targets: %w

What it means

During Kuma SD initialization, targets are fetched synchronously once so the first scrape has data. If updateTargetsLabels(ctx) fails (network error, non-200 from the control plane, bad xDS/mesh response), initialization aborts, the client is stopped, and the error is wrapped with this message. It means the initial Kuma target discovery failed.

Source

Thrown at lib/promscrape/discovery/kuma/api.go:95

	cfg := &apiConfig{
		client:   client,
		clientID: clientID,
		apiPath:  apiPath,

		fetchErrors: metrics.GetOrCreateCounter(fmt.Sprintf(`promscrape_discovery_kuma_errors_total{type="fetch",url=%q}`, sdc.Server)),
		parseErrors: metrics.GetOrCreateCounter(fmt.Sprintf(`promscrape_discovery_kuma_errors_total{type="parse",url=%q}`, sdc.Server)),
	}

	ctx, cancel := context.WithCancel(context.Background())
	cfg.cancel = cancel

	// Initialize targets synchronously and then start updating them in background.
	// The synchronous targets' update is needed for returning non-empty list of targets
	// just after the initialization.
	if err := cfg.updateTargetsLabels(ctx); err != nil {
		client.Stop()
		return nil, fmt.Errorf("cannot discover Kuma targets: %w", err)
	}
	cfg.wg.Go(func() {
		cfg.runTargetsWatcher(ctx)
	})

	return cfg, nil
}

func getAPIServerPath(serverURL string) (string, string, error) {
	if serverURL == "" {
		return "", "", fmt.Errorf("missing server url")
	}
	if !strings.Contains(serverURL, "://") {
		serverURL = "http://" + serverURL
	}
	psu, err := url.Parse(serverURL)
	if err != nil {
		return "", "", fmt.Errorf("cannot parse server url=%q: %w", serverURL, err)

View on GitHub (pinned to 5079fb58f1)

Solutions

1. Verify the Kuma control plane address/port is reachable from vmagent (curl http://kuma-cp:5676) and fix the `server` value
2. Check the wrapped error to distinguish connection refused/DNS/TLS from bad response data
3. Ensure startup ordering (vmagent starts after the control plane) or retry deployment
4. Inspect control-plane TLS certs and vmagent's CA configuration if handshake errors appear

Example fix

# before: vmagent starts before kuma-cp exists
server: "http://kuma-control-plane:5676"
# after: add readiness gating / correct address
server: "http://kuma-control-plane.kuma-system.svc:5676"

addr := strings.TrimPrefix(strings.TrimPrefix(server, "http://"), "https://")
conn, err := net.DialTimeout("tcp", addr, 3*time.Second)
if err != nil {
    return fmt.Errorf("kuma control plane %s not reachable: %w", addr, err)
}
conn.Close()

for attempt := 0; attempt < 5; attempt++ {
    ac, err := kuma.NewSDConfig(sdc)
    if err == nil {
        return ac, nil
    }
    if strings.Contains(err.Error(), "cannot discover Kuma targets") {
        time.Sleep(time.Duration(attempt+1) * 2 * time.Second)
        continue
    }
    return nil, err
}
return nil, errors.New("kuma control plane unreachable after retries")

When it happens

Trigger: Kuma control plane unreachable (DNS failure, connection refused, TLS handshake error) or returning invalid data when vmagent starts with a kuma_sd_config.

Common situations: Control plane not yet up during vmagent startup (ordering issue in k8s); wrong port; network policy/firewall blocking; expired control-plane mTLS certificates.

Related errors

• error when reading Kuma discovery response: %w
• cannot get API config for kuma_sd: %w
• cannot parse server %q: %w
• cannot create HTTP client for %q: %w
• missing server url

AI-assisted analysis of VictoriaMetrics/VictoriaMetrics@5079fb58f1 (2026-09-03). Data as JSON: /api/errors/b8f9bee5a379a1c3. Report an issue: GitHub.