{"record":{"id":"5a3f28cbb6077096","repo":"cilium/cilium","slug":"failed-to-collect-kubernetes-metrics-w","errorCode":null,"errorMessage":"failed to collect Kubernetes metrics: %w","messagePattern":"failed to collect Kubernetes metrics: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"cilium-cli/sysdump/sysdump.go","lineNumber":633,"sourceCode":"\t\t\tTask: func(ctx context.Context) error {\n\t\t\t\tn := corev1.NamespaceAll\n\t\t\t\tv, err := c.Client.ListUnstructured(ctx, k8sLeases, &n, metav1.ListOptions{})\n\t\t\t\tif err != nil {\n\t\t\t\t\treturn fmt.Errorf(\"failed to collect Kubernetes leases: %w\", err)\n\t\t\t\t}\n\t\t\t\tif err := c.WriteYAML(kubernetesLeasesFileName, v); err != nil {\n\t\t\t\t\treturn fmt.Errorf(\"failed to collect Kubernetes leases: %w\", err)\n\t\t\t\t}\n\t\t\t\treturn nil\n\t\t\t},\n\t\t},\n\t\t{\n\t\t\tDescription: \"Collecting Kubernetes metrics\",\n\t\t\tQuick:       true,\n\t\t\tTask: func(ctx context.Context) error {\n\t\t\t\tresult, err := c.Client.GetRaw(ctx, \"/metrics\")\n\t\t\t\tif err != nil {\n\t\t\t\t\treturn fmt.Errorf(\"failed to collect Kubernetes metrics: %w\", err)\n\t\t\t\t}\n\t\t\t\tif err := c.WriteString(kubernetesMetricsFileName, result); err != nil {\n\t\t\t\t\treturn fmt.Errorf(\"failed to collect Kubernetes metrics: %w\", err)\n\t\t\t\t}\n\t\t\t\treturn nil\n\t\t\t},\n\t\t},\n\t\t{\n\t\t\tDescription: \"Collecting Kubernetes nodes memory/cpu usage\",\n\t\t\tQuick:       true,\n\t\t\tTask: func(ctx context.Context) error {\n\t\t\t\t// Use the raw client to get the table format directly from the metrics API\n\t\t\t\t// This gives us the same output as kubectl top nodes\n\t\t\t\tresult, err := c.Client.GetRaw(ctx, \"/apis/metrics.k8s.io/v1beta1/nodes\")\n\t\t\t\tif err != nil {\n\t\t\t\t\treturn fmt.Errorf(\"failed to collect Kubernetes nodes memory/cpu usage: %w\", err)\n\t\t\t\t}\n","sourceCodeStart":615,"sourceCodeEnd":651,"githubUrl":"https://github.com/cilium/cilium/blob/ac7b90affa4baf0642e6685319d56907b3a73a6d/cilium-cli/sysdump/sysdump.go#L615-L651","documentation":"This error is raised by the 'Collecting Kubernetes metrics' sysdump task when Client.GetRaw(ctx, \"/metrics\") fails to fetch the kube-apiserver /metrics endpoint. It wraps the HTTP/API error with %w, so causes like 403 Forbidden on the metrics non-resource URL, disabled metrics endpoints, or API-server connectivity problems are preserved. It exists to label this specific collection step's failure within the sysdump's task log.","triggerScenarios":"c.Client.GetRaw(ctx, \"/metrics\") errors: the API server returns 403 for the non-resource URL /metrics, --enable-metrics... / authorization config blocks anonymous or SA access, the request times out, or the connection to the API server fails. (The WriteString variant of this message is error 814.)","commonSituations":"Hardened clusters where /metrics is RBAC-restricted to monitoring roles; kube-apiserver started with metrics disabled or behind an aggregation layer; short request timeouts on large, slow metric responses; corporate proxies stripping the path.","solutions":["Test the endpoint directly: 'kubectl get --raw /metrics' — replicate the 403/timeout with kubectl first.","Grant access to the non-resource URL: add nonResourceURLs: [\"/metrics\"] with verb get to the ClusterRole for the dumping identity.","Check kube-apiserver flags/authorization config to confirm the metrics endpoint is enabled and reachable.","If the response is merely slow, increase client timeout or retry the sysdump during lower API-server load."],"exampleFix":"// before\nError from server (Forbidden): \"/metrics\" is forbidden\n\n// after\nkubectl create clusterrole metrics-reader --verb=get --nonResourceURL=\"/metrics\"\nkubectl create clusterrolebinding metrics-reader-binding --clusterrole=metrics-reader --user=<user>","handlingStrategy":"try-catch","validationCode":"// verify /metrics access before the sysdump\nout, err := exec.Command(\"kubectl\", \"get\", \"--raw\", \"/metrics\").CombinedOutput()\nif err != nil {\n  return fmt.Errorf(\"pre-flight: /metrics not accessible: %v: %s\", err, out)\n}","typeGuard":"func isMetricsForbidden(err error) bool {\n  return strings.Contains(err.Error(), \"/metrics\") && strings.Contains(err.Error(), \"forbidden\")\n}","tryCatchPattern":"if err := collectMetrics(ctx); err != nil {\n  if isMetricsForbidden(err) {\n    log.Printf(\"sysdump: skipping metrics (RBAC on nonResourceURL /metrics): %v\", err)\n  } else {\n    return fmt.Errorf(\"metrics collection failed: %w\", err)\n  }\n}","preventionTips":["Grant get on nonResourceURLs [\"/metrics\"] in a ClusterRole for the dumping identity.","Smoke-test with 'kubectl get --raw /metrics' in the same identity/context.","On hardened clusters, expect /metrics to be blocked and treat it as a non-fatal skip.","Increase client timeouts if large metric payloads routinely time out."],"tags":["kubernetes","sysdump","metrics","rbac","cilium-cli"],"backgroundTag":"k8s-metrics-forbidden","analyzedSha":"ac7b90affa4baf0642e6685319d56907b3a73a6d","analyzedAt":"2026-08-31T18:27:15.868Z","schemaVersion":2},"datasetVersion":"2026-08-31T19:17:28.585Z"}