{"record":{"id":"f04c9c6d8e0ca3a1","repo":"cilium/cilium","slug":"failed-to-collect-the-cilium-clustermesh-gops-stat","errorCode":null,"errorMessage":"failed to collect the Cilium clustermesh gops stats: %w","messagePattern":"failed to collect the Cilium clustermesh gops stats: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"cilium-cli/sysdump/sysdump.go","lineNumber":1212,"sourceCode":"\t\t\t\tif err != nil {\n\t\t\t\t\treturn fmt.Errorf(\"failed to collect the Cilium clustermesh metrics: %w\", err)\n\t\t\t\t}\n\t\t\t\terr = c.SubmitMetricsSubtask(pods, defaults.ClusterMeshKVStoreMeshContainerName, defaults.ClusterMeshKVStoreMeshMetricsPortName)\n\t\t\t\tif err != nil {\n\t\t\t\t\treturn fmt.Errorf(\"failed to collect the Cilium clustermesh metrics: %w\", err)\n\t\t\t\t}\n\t\t\t\terr = c.SubmitMetricsSubtask(pods, defaults.ClusterMeshEtcdContainerName, defaults.ClusterMeshEtcdMetricsPortName)\n\t\t\t\tif err != nil {\n\t\t\t\t\treturn fmt.Errorf(\"failed to collect the Cilium clustermesh metrics: %w\", err)\n\t\t\t\t}\n\n\t\t\t\tfor container, port := range map[string]uint16{\n\t\t\t\t\tdefaults.ClusterMeshContainerName:            ciliumdef.GopsPortApiserver,\n\t\t\t\t\tdefaults.ClusterMeshKVStoreMeshContainerName: ciliumdef.GopsPortKVStoreMesh,\n\t\t\t\t} {\n\t\t\t\t\terr = c.SubmitGopsSubtasks(pods, container)\n\t\t\t\t\tif err != nil {\n\t\t\t\t\t\treturn fmt.Errorf(\"failed to collect the Cilium clustermesh gops stats: %w\", err)\n\t\t\t\t\t}\n\n\t\t\t\t\tif c.Options.Profiling {\n\t\t\t\t\t\terr = c.SubmitStreamProfilingGopsSubtasks(pods, container, port)\n\t\t\t\t\t\tif err != nil {\n\t\t\t\t\t\t\treturn fmt.Errorf(\"failed to collect the Cilium clustermesh profiles: %w\", err)\n\t\t\t\t\t\t}\n\t\t\t\t\t}\n\t\t\t\t}\n\n\t\t\t\treturn nil\n\t\t\t},\n\t\t},\n\t\t{\n\t\t\tDescription: \"Collecting the 'clustermesh-apiserver' deployment\",\n\t\t\tQuick:       true,\n\t\t\tTask: func(ctx context.Context) error {\n\t\t\t\tv, err := c.Client.GetDeployment(ctx, c.Options.CiliumNamespace, clustermeshApiserverDeploymentName, metav1.GetOptions{})","sourceCodeStart":1194,"sourceCodeEnd":1230,"githubUrl":"https://github.com/cilium/cilium/blob/ac7b90affa4baf0642e6685319d56907b3a73a6d/cilium-cli/sysdump/sysdump.go#L1194-L1230","documentation":"This error wraps failure of SubmitGopsSubtasks(pods, container) while Collector.Run collects gops stats for the clustermesh containers (clustermesh-apiserver, kvstoremesh). Gops stats are gathered by exec'ing the gops binary against a well-known gops port inside the container; failure means exec could not run or the gops agent/port is unavailable in that container. It affects only debug-data completeness of the sysdump.","triggerScenarios":"Collector.Run iterates the container/port map and calls SubmitGopsSubtasks for defaults.ClusterMeshContainerName or defaults.ClusterMeshKVStoreMeshContainerName; the submitted exec tasks fail because the container is not running, gops is not installed/binary name changed, or exec into the pod is denied.","commonSituations":"Container in CrashLoopBackOff so exec fails; minimal container images without gops; RBAC lacking pods/exec; Cilium image variants where the gops agent isn't listening; pod restarting during the sysdump.","solutions":["Ensure clustermesh pods are Running and ready before running the sysdump","Verify pods/exec RBAC for the kubeconfig identity","Confirm the gops binary exists in the container image for your Cilium version (kubectl exec <pod> -c <container> -- which gops)","Run with --debug to expose the wrapped exec error and retry","Skip gops collection (or ignore the error) if only basic sysdump artifacts are needed"],"exampleFix":"// before: hard failure aborts remaining sysdump tasks\nerr = c.SubmitGopsSubtasks(pods, container)\nif err != nil {\n    return fmt.Errorf(\"failed to collect the Cilium clustermesh gops stats: %w\", err)\n}\n// after: log and continue so the rest of the sysdump is preserved\nif err := c.SubmitGopsSubtasks(pods, container); err != nil {\n    c.logWarn(\"failed to collect gops stats for %q: %v\", container, err)\n}","handlingStrategy":"try-catch","validationCode":"for container := range map[string]uint16{\n    defaults.ClusterMeshContainerName:            ciliumdef.GopsPortApiserver,\n    defaults.ClusterMeshKVStoreMeshContainerName: ciliumdef.GopsPortKVStoreMesh,\n} {\n    for _, p := range AllPods(pods) {\n        if !podIsRunningAndHasContainer(p, container) {\n            continue\n        }\n        // gops must exist in the image for stats collection to work\n        fmt.Printf(\"gops collection target ready: %s/%s\\n\", p.Name, container)\n    }\n}","typeGuard":"func podReadyForGops(pod *corev1.Pod, container string) bool {\n    return podIsRunningAndHasContainer(pod, container)\n}","tryCatchPattern":"err := c.SubmitGopsSubtasks(pods, container)\nif err != nil {\n    c.logWarn(\"failed to collect the Cilium clustermesh gops stats for %q (continuing sysdump): %v\", container, err)\n    continue\n}","preventionTips":["Wait for clustermesh pods to be Running/Ready before the sysdump","Grant pods/exec RBAC to the cilium-cli identity","Verify gops is present in the container image (exec 'which gops')","Avoid running sysdump while pods are being rolled/restarted","Treat gops collection as best-effort: log and continue rather than aborting the sysdump"],"tags":["kubernetes","cilium","sysdump","gops","pod-exec"],"backgroundTag":"kubectl-exec-failed","analyzedSha":"ac7b90affa4baf0642e6685319d56907b3a73a6d","analyzedAt":"2026-08-31T18:27:15.868Z","schemaVersion":2},"datasetVersion":"2026-08-31T19:17:28.585Z"}