{"record":{"id":"96becdadafd2b266","repo":"cilium/cilium","slug":"failed-to-collect-profiling-data-from-cilium-pods","errorCode":null,"errorMessage":"failed to collect profiling data from Cilium pods: %w","messagePattern":"failed to collect profiling data from Cilium pods: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"cilium-cli/sysdump/sysdump.go","lineNumber":1612,"sourceCode":"\t\t})\n\t}\n\n\t// TODO(#2645): Ideally we would split ciliumTasks into\n\t// operator, agent, generic tasks, ..., etc\n\t// and only run then when feasible.\n\tif c.Options.CiliumNamespace != \"\" || c.Options.CiliumOperatorNamespace != \"\" {\n\t\ttasks = append(tasks, ciliumTasks...)\n\n\t\tserialTasks = append(serialTasks, Task{\n\t\t\tCreatesSubtasks: true,\n\t\t\tDescription:     \"Collecting profiling data from Cilium pods\",\n\t\t\tQuick:           false,\n\t\t\tTask: func(_ context.Context) error {\n\t\t\t\tif !c.Options.Profiling {\n\t\t\t\t\treturn nil\n\t\t\t\t}\n\t\t\t\tif err := c.SubmitProfilingGopsSubtasks(c.CiliumPods, ciliumAgentContainerName); err != nil {\n\t\t\t\t\treturn fmt.Errorf(\"failed to collect profiling data from Cilium pods: %w\", err)\n\t\t\t\t}\n\t\t\t\treturn nil\n\t\t\t},\n\t\t}, Task{\n\t\t\tCreatesSubtasks: true,\n\t\t\tDescription:     \"Collecting tracing data from Cilium pods\",\n\t\t\tQuick:           false,\n\t\t\tTask: func(_ context.Context) error {\n\t\t\t\tif !c.Options.Tracing {\n\t\t\t\t\treturn nil\n\t\t\t\t}\n\t\t\t\tif err := c.SubmitTracingGopsSubtask(c.CiliumPods, ciliumAgentContainerName); err != nil {\n\t\t\t\t\treturn fmt.Errorf(\"failed to collect tracing data from Cilium pods: %w\", err)\n\t\t\t\t}\n\t\t\t\treturn nil\n\t\t\t},\n\t\t})\n\t}","sourceCodeStart":1594,"sourceCodeEnd":1630,"githubUrl":"https://github.com/cilium/cilium/blob/ac7b90affa4baf0642e6685319d56907b3a73a6d/cilium-cli/sysdump/sysdump.go#L1594-L1630","documentation":"When c.Options.Profiling is enabled, the sysdump submits gops profiling subtasks against the Cilium agent container on each pod via c.SubmitProfilingGopsSubtasks. Any error from that submission is wrapped with this prefix and returned. It means profiling data could not be collected from the Cilium pods.","triggerScenarios":"c.Options.Profiling == true and c.SubmitProfilingGopsSubtasks(c.CiliumPods, ciliumAgentContainerName) returns an error: empty/invalid pod list, pod exec failures, or the gops agent not accepting profiling requests inside the container.","commonSituations":"Cilium agent pods not Running or crashing so exec fails; gops profiling not compiled/enabled in the agent image; restricted environments (PSP/OPA) blocking pod exec; running profiling collection against the wrong container name.","solutions":["Ensure Cilium agent pods are Running and exec into them works (kubectl exec -n <ns> <cilium-pod> -c cilium-agent true)","Confirm the cilium image has gops profiling support before passing --profiling","Retry the sysdump once the cluster is healthy","Read the wrapped (%w) cause in the sysdump output for the precise per-pod failure"],"exampleFix":"// before\ncilium-cli sysdump --profiling   # agents CrashLooping -> exec fails\n// after\nkubectl -n kube-system rollout status ds/cilium && cilium-cli sysdump --profiling","handlingStrategy":"try-catch","validationCode":"if opts.Profiling {\n    pods, _ := client.ListPods(ctx, ciliumNs, metav1.ListOptions{LabelSelector: \"k8s-app=cilium\"})\n    for _, p := range pods.Items {\n        if p.Status.Phase != corev1.PodRunning {\n            return fmt.Errorf(\"agent pod %s not Running; skip profiling collection\", p.Name)\n        }\n    }\n}","typeGuard":"func isProfilingCollectionErr(err error) bool {\n    return err != nil && strings.Contains(err.Error(), \"failed to collect profiling data\")\n}","tryCatchPattern":"if err := sysdump.Run(ctx); err != nil {\n    if strings.Contains(err.Error(), \"failed to collect profiling data\") {\n        log.Printf(\"profiling collection failed, continuing without it: %v\", unwrapAll(err))\n        return nil\n    }\n    return err\n}","preventionTips":["Only pass --profiling on images built with gops support","Verify agent pods are Running and exec-capable (kubectl exec ... true) first","Do not enable profiling during unstable rollouts when pods restart frequently","Check RBAC allows pods/exec in the Cilium namespace"],"tags":["kubernetes","cilium","sysdump","profiling","gops"],"backgroundTag":"kubernetes-pod-exec-failed","analyzedSha":"ac7b90affa4baf0642e6685319d56907b3a73a6d","analyzedAt":"2026-08-31T18:27:15.868Z","schemaVersion":2},"datasetVersion":"2026-08-31T19:17:28.585Z"}