cilium/cilium · error

failed to collect gops for %q (%q) in namespace %q: %w

Error message

failed to collect gops for %q (%q) in namespace %q: %w

What it means

During gops stats collection, the collector execs 'gops <stat> <agentPID>' in each pod container through a file sink. If that exec fails, the error is wrapped with pod, container, and namespace. It is thrown inside the per-pod gops task in the gops collection subtask.

Source

Thrown at cilium-cli/sysdump/sysdump.go:2820

		if !podIsRunningAndHasContainer(p, containerName) {
			continue
		}

		for _, g := range gopsStats {
			if err := c.Pool.Submit(fmt.Sprintf("gops-%s-%s", p.Name, g), func(ctx context.Context) error {
				gopsCommand, agentPID, err := c.getGopsPID(ctx, p, containerName)
				if err != nil {
					return err
				}

				if err := c.WithFileSink(fmt.Sprintf(gopsFileName, p.Name, containerName, g), func(out io.Writer) error {
					return c.Client.ExecInPodWithWriters(ctx, nil, p.Namespace, p.Name, containerName, []string{
						gopsCommand,
						g,
						agentPID,
					}, out, k8s.StderrAsError)
				}); err != nil {
					return fmt.Errorf("failed to collect gops for %q (%q) in namespace %q: %w", p.Name, containerName, p.Namespace, err)
				}

				return nil
			}); err != nil {
				return fmt.Errorf("failed to submit %s gops task for %q: %w", g, p.Name, err)
			}
		}
	}
	return nil
}

// SubmitProfilingGopsSubtasks submits tasks to collect profiling data from pods.
func (c *Collector) SubmitProfilingGopsSubtasks(pods []*corev1.Pod, containerName string) error {
	for _, p := range pods {
		for g := range gopsProfiling {
			if err := c.Pool.Submit(fmt.Sprintf("gops-%s-%s", p.Name, g), func(ctx context.Context) error {
				gopsCommand, agentPID, err := c.getGopsPID(ctx, p, containerName)
				if err != nil {

View on GitHub (pinned to ac7b90affa)

Solutions

1. Inspect the wrapped error: 'command not found' means gops is absent; exit errors often mean an invalid PID or stat
2. Re-run so the PID is re-detected after any container restart
3. Ensure the cilium/cilium-node image embeds the gops agent (it must be built in for gops to work)
4. Update cilium-cli so the gops stat list matches the agent's gops version

Example fix

// before
return fmt.Errorf("failed to collect gops for %q (%q) in namespace %q: %w", p.Name, containerName, p.Namespace, err)
// after: skip cleanly when gops is unavailable
if execErr, ok := err.(*exec.ExitError); ok && strings.Contains(string(execErr.Stderr), "could not locate agent") {
    log.Printf("gops agent unavailable in %s, skipping stat %s", p.Name, g)
    return nil
}
return fmt.Errorf("failed to collect gops for %q (%q) in namespace %q: %w", p.Name, containerName, p.Namespace, err)

// re-detect PID right before gops exec to avoid stale values
cmd, pid, err := getGopsPID(ctx, pod, container)
if err != nil || pid == "" {
    return fmt.Errorf("no live gops agent PID in %s: %w", pod.Name, err)
}

func gopsUsable(pid string) bool {
    n, err := strconv.Atoi(pid)
    return err == nil && n > 0
}

if err := collector.collectGops(ctx, pods); err != nil {
    if strings.Contains(err.Error(), "could not locate agent") {
        log.Printf("gops agent not embedded in target binary; skipping")
        return nil
    }
    return err
}

When it happens

Trigger: ExecInPodWithWriters(ctx, nil, p.Namespace, p.Name, containerName, [gopsCommand, g, agentPID], out, k8s.StderrAsError) returns an error — bad agentPID, gops binary missing, or stderr output treated as an error.

Common situations: Stale/wrong gops PID (agent not enabled in the cilium binary); gops stat name unsupported by the in-container gops version; container restarted between PID discovery and profiling; exec blocked by policy.

Related errors

• failed to list processes %q (%q) in namespace %q: %w
• failed to collect the Cilium clustermesh gops stats: %w
• failed to submit SPIRE entries task for %q: %w
• failed to extract pid from output: %q
• Unable to find output file: %s

AI-assisted analysis of cilium/cilium@ac7b90affa (2026-08-31). Data as JSON: /api/errors/7214524cc49e37db. Report an issue: GitHub.