cilium/cilium · warning

failed to collect the Cilium clustermesh gops stats: %w

Error message

failed to collect the Cilium clustermesh gops stats: %w

What it means

This error wraps failure of SubmitGopsSubtasks(pods, container) while Collector.Run collects gops stats for the clustermesh containers (clustermesh-apiserver, kvstoremesh). Gops stats are gathered by exec'ing the gops binary against a well-known gops port inside the container; failure means exec could not run or the gops agent/port is unavailable in that container. It affects only debug-data completeness of the sysdump.

Source

Thrown at cilium-cli/sysdump/sysdump.go:1212

				if err != nil {
					return fmt.Errorf("failed to collect the Cilium clustermesh metrics: %w", err)
				}
				err = c.SubmitMetricsSubtask(pods, defaults.ClusterMeshKVStoreMeshContainerName, defaults.ClusterMeshKVStoreMeshMetricsPortName)
				if err != nil {
					return fmt.Errorf("failed to collect the Cilium clustermesh metrics: %w", err)
				}
				err = c.SubmitMetricsSubtask(pods, defaults.ClusterMeshEtcdContainerName, defaults.ClusterMeshEtcdMetricsPortName)
				if err != nil {
					return fmt.Errorf("failed to collect the Cilium clustermesh metrics: %w", err)
				}

				for container, port := range map[string]uint16{
					defaults.ClusterMeshContainerName:            ciliumdef.GopsPortApiserver,
					defaults.ClusterMeshKVStoreMeshContainerName: ciliumdef.GopsPortKVStoreMesh,
				} {
					err = c.SubmitGopsSubtasks(pods, container)
					if err != nil {
						return fmt.Errorf("failed to collect the Cilium clustermesh gops stats: %w", err)
					}

					if c.Options.Profiling {
						err = c.SubmitStreamProfilingGopsSubtasks(pods, container, port)
						if err != nil {
							return fmt.Errorf("failed to collect the Cilium clustermesh profiles: %w", err)
						}
					}
				}

				return nil
			},
		},
		{
			Description: "Collecting the 'clustermesh-apiserver' deployment",
			Quick:       true,
			Task: func(ctx context.Context) error {
				v, err := c.Client.GetDeployment(ctx, c.Options.CiliumNamespace, clustermeshApiserverDeploymentName, metav1.GetOptions{})

View on GitHub (pinned to ac7b90affa)

Solutions

1. Ensure clustermesh pods are Running and ready before running the sysdump
2. Verify pods/exec RBAC for the kubeconfig identity
3. Confirm the gops binary exists in the container image for your Cilium version (kubectl exec <pod> -c <container> -- which gops)
4. Run with --debug to expose the wrapped exec error and retry
5. Skip gops collection (or ignore the error) if only basic sysdump artifacts are needed

Example fix

// before: hard failure aborts remaining sysdump tasks
err = c.SubmitGopsSubtasks(pods, container)
if err != nil {
    return fmt.Errorf("failed to collect the Cilium clustermesh gops stats: %w", err)
}
// after: log and continue so the rest of the sysdump is preserved
if err := c.SubmitGopsSubtasks(pods, container); err != nil {
    c.logWarn("failed to collect gops stats for %q: %v", container, err)
}

for container := range map[string]uint16{
    defaults.ClusterMeshContainerName:            ciliumdef.GopsPortApiserver,
    defaults.ClusterMeshKVStoreMeshContainerName: ciliumdef.GopsPortKVStoreMesh,
} {
    for _, p := range AllPods(pods) {
        if !podIsRunningAndHasContainer(p, container) {
            continue
        }
        // gops must exist in the image for stats collection to work
        fmt.Printf("gops collection target ready: %s/%s\n", p.Name, container)
    }
}

func podReadyForGops(pod *corev1.Pod, container string) bool {
    return podIsRunningAndHasContainer(pod, container)
}

err := c.SubmitGopsSubtasks(pods, container)
if err != nil {
    c.logWarn("failed to collect the Cilium clustermesh gops stats for %q (continuing sysdump): %v", container, err)
    continue
}

When it happens

Trigger: Collector.Run iterates the container/port map and calls SubmitGopsSubtasks for defaults.ClusterMeshContainerName or defaults.ClusterMeshKVStoreMeshContainerName; the submitted exec tasks fail because the container is not running, gops is not installed/binary name changed, or exec into the pod is denied.

Common situations: Container in CrashLoopBackOff so exec fails; minimal container images without gops; RBAC lacking pods/exec; Cilium image variants where the gops agent isn't listening; pod restarting during the sysdump.

Related errors

• failed to collect the Cilium clustermesh debug information:
• failed to collect Cilium gops: %w
• failed to collect cilium-operator gops stats: %w
• failed to collect Hubble gops: %w
• failed to collect hubble flows: %w

AI-assisted analysis of cilium/cilium@ac7b90affa (2026-08-31). Data as JSON: /api/errors/f04c9c6d8e0ca3a1. Report an issue: GitHub.