{"record":{"id":"7214524cc49e37db","repo":"cilium/cilium","slug":"failed-to-collect-gops-for-q-q-in-namespace-q","errorCode":null,"errorMessage":"failed to collect gops for %q (%q) in namespace %q: %w","messagePattern":"failed to collect gops for %q \\(%q\\) in namespace %q: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"cilium-cli/sysdump/sysdump.go","lineNumber":2820,"sourceCode":"\t\tif !podIsRunningAndHasContainer(p, containerName) {\n\t\t\tcontinue\n\t\t}\n\n\t\tfor _, g := range gopsStats {\n\t\t\tif err := c.Pool.Submit(fmt.Sprintf(\"gops-%s-%s\", p.Name, g), func(ctx context.Context) error {\n\t\t\t\tgopsCommand, agentPID, err := c.getGopsPID(ctx, p, containerName)\n\t\t\t\tif err != nil {\n\t\t\t\t\treturn err\n\t\t\t\t}\n\n\t\t\t\tif err := c.WithFileSink(fmt.Sprintf(gopsFileName, p.Name, containerName, g), func(out io.Writer) error {\n\t\t\t\t\treturn c.Client.ExecInPodWithWriters(ctx, nil, p.Namespace, p.Name, containerName, []string{\n\t\t\t\t\t\tgopsCommand,\n\t\t\t\t\t\tg,\n\t\t\t\t\t\tagentPID,\n\t\t\t\t\t}, out, k8s.StderrAsError)\n\t\t\t\t}); err != nil {\n\t\t\t\t\treturn fmt.Errorf(\"failed to collect gops for %q (%q) in namespace %q: %w\", p.Name, containerName, p.Namespace, err)\n\t\t\t\t}\n\n\t\t\t\treturn nil\n\t\t\t}); err != nil {\n\t\t\t\treturn fmt.Errorf(\"failed to submit %s gops task for %q: %w\", g, p.Name, err)\n\t\t\t}\n\t\t}\n\t}\n\treturn nil\n}\n\n// SubmitProfilingGopsSubtasks submits tasks to collect profiling data from pods.\nfunc (c *Collector) SubmitProfilingGopsSubtasks(pods []*corev1.Pod, containerName string) error {\n\tfor _, p := range pods {\n\t\tfor g := range gopsProfiling {\n\t\t\tif err := c.Pool.Submit(fmt.Sprintf(\"gops-%s-%s\", p.Name, g), func(ctx context.Context) error {\n\t\t\t\tgopsCommand, agentPID, err := c.getGopsPID(ctx, p, containerName)\n\t\t\t\tif err != nil {","sourceCodeStart":2802,"sourceCodeEnd":2838,"githubUrl":"https://github.com/cilium/cilium/blob/ac7b90affa4baf0642e6685319d56907b3a73a6d/cilium-cli/sysdump/sysdump.go#L2802-L2838","documentation":"During gops stats collection, the collector execs 'gops <stat> <agentPID>' in each pod container through a file sink. If that exec fails, the error is wrapped with pod, container, and namespace. It is thrown inside the per-pod gops task in the gops collection subtask.","triggerScenarios":"ExecInPodWithWriters(ctx, nil, p.Namespace, p.Name, containerName, [gopsCommand, g, agentPID], out, k8s.StderrAsError) returns an error — bad agentPID, gops binary missing, or stderr output treated as an error.","commonSituations":"Stale/wrong gops PID (agent not enabled in the cilium binary); gops stat name unsupported by the in-container gops version; container restarted between PID discovery and profiling; exec blocked by policy.","solutions":["Inspect the wrapped error: 'command not found' means gops is absent; exit errors often mean an invalid PID or stat","Re-run so the PID is re-detected after any container restart","Ensure the cilium/cilium-node image embeds the gops agent (it must be built in for gops to work)","Update cilium-cli so the gops stat list matches the agent's gops version"],"exampleFix":"// before\nreturn fmt.Errorf(\"failed to collect gops for %q (%q) in namespace %q: %w\", p.Name, containerName, p.Namespace, err)\n// after: skip cleanly when gops is unavailable\nif execErr, ok := err.(*exec.ExitError); ok && strings.Contains(string(execErr.Stderr), \"could not locate agent\") {\n    log.Printf(\"gops agent unavailable in %s, skipping stat %s\", p.Name, g)\n    return nil\n}\nreturn fmt.Errorf(\"failed to collect gops for %q (%q) in namespace %q: %w\", p.Name, containerName, p.Namespace, err)","handlingStrategy":"try-catch","validationCode":"// re-detect PID right before gops exec to avoid stale values\ncmd, pid, err := getGopsPID(ctx, pod, container)\nif err != nil || pid == \"\" {\n    return fmt.Errorf(\"no live gops agent PID in %s: %w\", pod.Name, err)\n}","typeGuard":"func gopsUsable(pid string) bool {\n    n, err := strconv.Atoi(pid)\n    return err == nil && n > 0\n}","tryCatchPattern":"if err := collector.collectGops(ctx, pods); err != nil {\n    if strings.Contains(err.Error(), \"could not locate agent\") {\n        log.Printf(\"gops agent not embedded in target binary; skipping\")\n        return nil\n    }\n    return err\n}","preventionTips":["Detect the gops PID immediately before each collection round","Confirm the gops agent is compiled into the cilium binary","Refresh PID after any container restart","Keep cilium-cli gops stats aligned with agent gops version"],"tags":["cilium-cli","sysdump","gops","pod-exec"],"backgroundTag":"pod-exec-failure","analyzedSha":"ac7b90affa4baf0642e6685319d56907b3a73a6d","analyzedAt":"2026-08-31T18:27:15.868Z","schemaVersion":2},"datasetVersion":"2026-08-31T22:30:34.772Z"}