What causes "error running %q with args %q: %w" in cilium/cilium?

Read the wrapped error and the CombinedOutput text (logged at warn) to see the exact `ip` failure.

error running %q with args %q: %w

Error message

error running %q with args %q: %w

What it means

configureHealthRouting runs each rendered `ip` command inside the cilium-health netns with exec.Command and wraps any non-zero/execution error with the program and its arguments. It means the kernel/iproute2 rejected a route configuration step for the health endpoint.

Source

Thrown at pkg/health/health_connectivity_endpoint.go:135

// configureHealthRouting is meant to be run inside the health service netns
func (h *ciliumHealthManager) configureHealthRouting(routes []route.Route, dev string) error {
	for _, rt := range routes {
		cmd := rt.ToIPCommand(dev)
		if len(cmd) < 2 {
			return fmt.Errorf("ip command %s not expected len!", cmd)
		}
		prog := cmd[0]
		args := cmd[1:]
		h.logger.Debug(fmt.Sprintf("Running \"%s %+v\"", prog, args))
		out, err := exec.Command(prog, args...).CombinedOutput()
		if err == nil && len(out) > 0 {
			h.logger.Warn(string(out),
				logfields.Prog, prog,
				logfields.Args, args,
			)
		} else if err != nil {
			return fmt.Errorf("error running %q with args %q: %w", prog, args, err)
		}
	}
	return nil
}

// configureHealthInterface is meant to be run inside the health service netns
func (h *ciliumHealthManager) configureHealthInterface(ifName string, ip4Addr, ip6Addr *net.IPNet) error {
	link, err := safenetlink.LinkByName(ifName)
	if err != nil {
		return err
	}

	if ip6Addr == nil {
		// Use the direct sysctl without reconciliation of errors since we're in a different
		// network namespace and thus can't use the normal sysctl API.
		sysctl := sysctl.NewDirectSysctl(afero.NewOsFs(), option.Config.ProcFs)
		// Ignore the error; if IPv6 is completely disabled
		// then it's okay if we can't write the sysctl.

View on GitHub (pinned to ac7b90affa)

Solutions

Read the wrapped error and the CombinedOutput text (logged at warn) to see the exact `ip` failure.
Ensure the process has CAP_NET_ADMIN/CAP_SYS_ADMIN to configure routes in the netns.
Verify the target device (dev argument) exists inside the health netns (`ip netns exec cilium-health ip link`).
If 'file exists' / 'RTNETLINK answers: File exists', flush stale routes in the health netns and retry.

Example fix

// before: error surfaces as 'error running "ip" with args ["-6" "route" "add" ...]: exit status 2'
// after: capture output for diagnosis
out, err := exec.Command(prog, args...).CombinedOutput()
if err != nil {
    return fmt.Errorf("error running %q with args %q: %w: %s", prog, args, err, string(out))
}

Defensive patterns

Strategy: try-catch

Validate before calling

// pre-check device exists in health netns
if err := exec.Command("ip", "netns", "exec", "cilium-health", "ip", "link", "show", dev).Run(); err != nil {
    // fix datapath setup before configuring routes
}

Try / catch

out, err := exec.Command(prog, args...).CombinedOutput()
if err != nil {
    if strings.Contains(string(out), "File exists") {
        // flush duplicate route and retry once
    }
    return fmt.Errorf("ip %v failed: %w: %s", args, err, out)
}

Prevention

Run the agent with CAP_NET_ADMIN and CAP_SYS_ADMIN.
Clean up stale netns/routes from previous runs before configuring.
Always log CombinedOutput alongside the error for iproute2 diagnostics.

When it happens

Trigger: exec.Command(prog, args...).CombinedOutput() returns err — e.g. `ip route add` fails because the device doesn't exist in the netns, permission denied (missing CAP_NET_ADMIN), or the route already exists / prefix is unreachable.

Common situations: Running cilium-health with dropped capabilities; MTU/route arguments rejected by the kernel; duplicate route insertion after a partial retry; SELinux/AppArmor blocking ip inside the namespace.

Related errors

AI-assisted analysis of cilium/cilium@ac7b90affa (2026-08-31). Data as JSON: /api/errors/c603b930225e228d. Report an issue: GitHub.