cilium/cilium · warning

failed to submit metrics task for %q: %w

Error message

failed to submit metrics task for %q: %w

What it means

Outer wrapped error from the cilium-cli sysdump when submitting the per-pod metrics collection task to the worker pool fails, or when the task body returned any of the inner metrics errors (1045-1047). The pod name is included for context.

Source

Thrown at cilium-cli/sysdump/sysdump.go:3130

		if !podIsRunningAndHasContainer(p, containerName) {
			continue
		}
		err := c.Pool.Submit(fmt.Sprintf("metrics-%s-%s-%s", p.Name, containerName, portName), func(ctx context.Context) error {
			port, err := getPodMetricsPort(p, containerName, portName)
			if err != nil {
				return fmt.Errorf("failed to collect metrics: %w - this is expected if prometheus metrics are disabled", err)
			}
			rsp, err := c.Client.ProxyGet(ctx, p.Namespace, fmt.Sprintf("%s:%d", p.Name, port), "metrics")
			if err != nil {
				return fmt.Errorf("failed to collect metrics: %w", err)
			}
			if err := c.WriteString(fmt.Sprintf(metricsFileName, p.Name, containerName), rsp); err != nil {
				return fmt.Errorf("failed to collect metrics: %w", err)
			}
			return nil
		})
		if err != nil {
			return fmt.Errorf("failed to submit metrics task for %q: %w", p.Name, err)
		}
	}
	return nil
}

func (c *Collector) submitCiliumOperatorDbgTasks(pods []*corev1.Pod) error {
	tasks := []struct {
		name string
		ext  string
		args []string
	}{
		{
			name: "statedb-dump",
			ext:  "json",
			args: []string{"shell", "db/dump"},
		},
		{
			name: "status-clustermesh",

View on GitHub (pinned to ac7b90affa)

Solutions

1. Unwrap to find the inner cause (port lookup vs scrape vs write)
2. If the cause is the 'expected if prometheus metrics are disabled' error, enable metrics or ignore
3. Fix RBAC/disk issues identified in the wrapped error
4. Re-run cilium-cli sysdump

Example fix

// before
return fmt.Errorf("failed to submit metrics task for %q: %w", p.Name, err)
// after
if isExpectedMetricsErr(err) {
    c.logDebug("metrics collection skipped for pod %s: %v", p.Name, err)
    return nil
}
return fmt.Errorf("failed to submit metrics task for %q: %w", p.Name, err)

// skip submission when the container has no metrics port
if !hasMetricsPort(p, containerName, portName) { continue }

func isExpectedMetricsErr(err error) bool {
    return strings.Contains(err.Error(), "expected if prometheus metrics are disabled")
}

if err := c.Pool.Submit(taskName, task); err != nil {
    if isExpectedMetricsErr(err) {
        c.logDebug("metrics skipped for pod %s: %v", p.Name, err)
        continue
    }
    return fmt.Errorf("failed to submit metrics task for %q: %w", p.Name, err)
}

When it happens

Trigger: After c.Pool.Submit("metrics-<pod>-<container>-<port>", ...) returns non-nil, the collector wraps it with this message; the cause may be a port-lookup, ProxyGet, or WriteString failure.

Common situations: Sysdump runs where one agent pod's metrics collection failed; because tasks run concurrently, several pods can report this independently.

Related errors

• failed to collect Kubernetes metrics: %w
• submit %q task: %w
• failed to collect metrics: %w - this is expected if promethe
• failed to collect metrics: %w
• failed to create sysdump collector: %w

AI-assisted analysis of cilium/cilium@ac7b90affa (2026-08-31). Data as JSON: /api/errors/08bf123288dcee50. Report an issue: GitHub.