cilium/cilium · error

failed to get logs from Hubble Relay pods

Error message

failed to get logs from Hubble Relay pods

What it means

The sysdump task for Hubble Relay logs lists pods using HubbleRelayLabelSelector in the Cilium namespace. A failure of the ListPods API call is reported as this static message with the real cause discarded. It means Hubble Relay pods could not be enumerated, so their logs were not collected.

Source

Thrown at cilium-cli/sysdump/sysdump.go:1516

				if err != nil {
					return fmt.Errorf("failed to get logs from Hubble pods")
				}
				if err := c.SubmitLogsTasks(FilterPods(p, c.NodeList), c.Options.LogsSinceTime, c.Options.LogsLimitBytes); err != nil {
					return fmt.Errorf("failed to collect logs from Hubble pods")
				}
				return nil
			},
		},
		{
			CreatesSubtasks: true,
			Description:     "Collecting logs from Hubble Relay pods",
			Quick:           false,
			Task: func(ctx context.Context) error {
				p, err := c.Client.ListPods(ctx, c.Options.CiliumNamespace, metav1.ListOptions{
					LabelSelector: c.Options.HubbleRelayLabelSelector,
				})
				if err != nil {
					return fmt.Errorf("failed to get logs from Hubble Relay pods")
				}
				if err := c.SubmitLogsTasks(AllPods(p), c.Options.LogsSinceTime, c.Options.LogsLimitBytes); err != nil {
					return fmt.Errorf("failed to collect logs from Hubble Relay pods")
				}
				return nil
			},
		},
		{
			CreatesSubtasks: true,
			Description:     "Collecting logs from Hubble UI pods",
			Quick:           false,
			Task: func(ctx context.Context) error {
				p, err := c.Client.ListPods(ctx, c.Options.CiliumNamespace, metav1.ListOptions{
					LabelSelector: c.Options.HubbleUILabelSelector,
				})
				if err != nil {
					return fmt.Errorf("failed to get logs from Hubble UI pods")
				}

View on GitHub (pinned to ac7b90affa)

Solutions

1. Verify access: kubectl get pods -n <cilium-namespace> -l k8s-app=hubble-relay works.
2. Correct the --cilium-namespace flag to the namespace where Hubble Relay is deployed.
3. Check RBAC: kubectl auth can-i list pods -n <cilium-namespace>.
4. Retry the sysdump after restoring cluster connectivity.

Example fix

// before
return fmt.Errorf("failed to get logs from Hubble Relay pods")
// after
return fmt.Errorf("failed to get logs from Hubble Relay pods: %w", err)

pods, _ := clientset.CoreV1().Pods(ciliumNamespace).List(ctx, metav1.ListOptions{LabelSelector: hubbleSelector})
for _, p := range pods.Items {
    if p.Status.Phase != corev1.PodRunning {
        log.Warnf("Hubble pod %s not Running; log collection may fail", p.Name)
    }
}

if err := task(ctx); err != nil && strings.Contains(err.Error(), "failed to collect logs from Hubble pods") {
    log.Warn("Hubble log collection failed; check pod health, node filtering, and log limits")
}

When it happens

Trigger: c.Client.ListPods(ctx, c.Options.CiliumNamespace, metav1.ListOptions{LabelSelector: c.Options.HubbleRelayLabelSelector}) returns an error in the 'Collecting logs from Hubble Relay pods' task (sysdump.go:1512-1516).

Common situations: Hubble Relay not installed / namespace typo so label selector queries fail or RBAC blocks listing; expired kubeconfig or disconnected VPN; API server instability during sysdump runs.

Related errors

• failed to get Hubble Relay pods: %w
• failed to collect the Hubble Relay configuration: %w
• failed to collect the Hubble Relay deployment: %w
• failed to collect the Hubble Relay deployment %q: %w
• failed to get logs from the Cluster Mesh certgen pods

AI-assisted analysis of cilium/cilium@ac7b90affa (2026-08-31). Data as JSON: /api/errors/49650b85cb3723df. Report an issue: GitHub.