cilium/cilium · error

failed to collect Kubernetes metrics: %w

Error message

failed to collect Kubernetes metrics: %w

What it means

This error is raised by the 'Collecting Kubernetes metrics' sysdump task when Client.GetRaw(ctx, "/metrics") fails to fetch the kube-apiserver /metrics endpoint. It wraps the HTTP/API error with %w, so causes like 403 Forbidden on the metrics non-resource URL, disabled metrics endpoints, or API-server connectivity problems are preserved. It exists to label this specific collection step's failure within the sysdump's task log.

Source

Thrown at cilium-cli/sysdump/sysdump.go:633

			Task: func(ctx context.Context) error {
				n := corev1.NamespaceAll
				v, err := c.Client.ListUnstructured(ctx, k8sLeases, &n, metav1.ListOptions{})
				if err != nil {
					return fmt.Errorf("failed to collect Kubernetes leases: %w", err)
				}
				if err := c.WriteYAML(kubernetesLeasesFileName, v); err != nil {
					return fmt.Errorf("failed to collect Kubernetes leases: %w", err)
				}
				return nil
			},
		},
		{
			Description: "Collecting Kubernetes metrics",
			Quick:       true,
			Task: func(ctx context.Context) error {
				result, err := c.Client.GetRaw(ctx, "/metrics")
				if err != nil {
					return fmt.Errorf("failed to collect Kubernetes metrics: %w", err)
				}
				if err := c.WriteString(kubernetesMetricsFileName, result); err != nil {
					return fmt.Errorf("failed to collect Kubernetes metrics: %w", err)
				}
				return nil
			},
		},
		{
			Description: "Collecting Kubernetes nodes memory/cpu usage",
			Quick:       true,
			Task: func(ctx context.Context) error {
				// Use the raw client to get the table format directly from the metrics API
				// This gives us the same output as kubectl top nodes
				result, err := c.Client.GetRaw(ctx, "/apis/metrics.k8s.io/v1beta1/nodes")
				if err != nil {
					return fmt.Errorf("failed to collect Kubernetes nodes memory/cpu usage: %w", err)
				}

View on GitHub (pinned to ac7b90affa)

Solutions

1. Test the endpoint directly: 'kubectl get --raw /metrics' — replicate the 403/timeout with kubectl first.
2. Grant access to the non-resource URL: add nonResourceURLs: ["/metrics"] with verb get to the ClusterRole for the dumping identity.
3. Check kube-apiserver flags/authorization config to confirm the metrics endpoint is enabled and reachable.
4. If the response is merely slow, increase client timeout or retry the sysdump during lower API-server load.

Example fix

// before
Error from server (Forbidden): "/metrics" is forbidden

// after
kubectl create clusterrole metrics-reader --verb=get --nonResourceURL="/metrics"
kubectl create clusterrolebinding metrics-reader-binding --clusterrole=metrics-reader --user=<user>

// verify /metrics access before the sysdump
out, err := exec.Command("kubectl", "get", "--raw", "/metrics").CombinedOutput()
if err != nil {
  return fmt.Errorf("pre-flight: /metrics not accessible: %v: %s", err, out)
}

func isMetricsForbidden(err error) bool {
  return strings.Contains(err.Error(), "/metrics") && strings.Contains(err.Error(), "forbidden")
}

if err := collectMetrics(ctx); err != nil {
  if isMetricsForbidden(err) {
    log.Printf("sysdump: skipping metrics (RBAC on nonResourceURL /metrics): %v", err)
  } else {
    return fmt.Errorf("metrics collection failed: %w", err)
  }
}

When it happens

Trigger: c.Client.GetRaw(ctx, "/metrics") errors: the API server returns 403 for the non-resource URL /metrics, --enable-metrics... / authorization config blocks anonymous or SA access, the request times out, or the connection to the API server fails. (The WriteString variant of this message is error 814.)

Common situations: Hardened clusters where /metrics is RBAC-restricted to monitoring roles; kube-apiserver started with metrics disabled or behind an aggregation layer; short request timeouts on large, slow metric responses; corporate proxies stripping the path.

Related errors

• failed to collect Kubernetes services: %w
• failed to collect Kubernetes network policies: %w
• failed to collect Kubernetes endpoints: %w
• failed to collect Kubernetes endpointslices: %w
• failed to collect Kubernetes leases: %w

AI-assisted analysis of cilium/cilium@ac7b90affa (2026-08-31). Data as JSON: /api/errors/5a3f28cbb6077096. Report an issue: GitHub.