cilium/cilium · critical

failed to add internal IP address to %s: %w

Error message

failed to add internal IP address to %s: %w

What it means

After sysctls are applied, Reinitialize assigns the node's internal IPv4/IPv6 addresses (router IPs) to the cilium_host device via addHostDeviceAddr. Failure to add either address is wrapped with the device name. Without the internal IPs on cilium_host, host-originated and routed pod traffic cannot work.

Source

Thrown at pkg/datapath/loader/base.go:374

	if option.Config.IPAM == ipamOption.IPAMENI {
		var err error
		if sysSettings, err = addIPv4ENIRules(l.logger, sysSettings); err != nil {
			return fmt.Errorf("unable to install ip rule for ENI multi-node NodePort: %w", err)
		}
		if err = addIPv6ENIRules(); err != nil {
			return fmt.Errorf("unable to install ipv6 ip rule for ENI multi-node NodePort: %w", err)
		}
	}

	// Any code that relies on sysctl settings being applied needs to be called after this.
	if err := l.sysctl.ApplySettings(sysSettings); err != nil {
		return err
	}

	// add internal ipv4 and ipv6 addresses to cilium_host
	if err := addHostDeviceAddr(hostDev1, internalIPv4, internalIPv6); err != nil {
		return fmt.Errorf("failed to add internal IP address to %s: %w", hostDev1.Attrs().Name, err)
	}

	devices := lnc.DeviceNames()
	if err := cleanIngressQdisc(l.logger, devices); err != nil {
		l.logger.Warn("Unable to clean up ingress qdiscs", logfields.Error, err)
		return err
	}

	if err := l.writeNodeConfigHeader(lnc); err != nil {
		l.logger.Error("Unable to write node config header", logfields.Error, err)
		return err
	}

	if err := l.writeNetdevHeader("./"); err != nil {
		l.logger.Warn("Unable to write netdev header", logfields.Error, err)
		return err
	}

View on GitHub (pinned to ac7b90affa)

Solutions

1. Check the wrapped inner error: 'File exists' indicates a conflicting address on cilium_host
2. Inspect 'ip addr show cilium_host' and remove stale/incorrect addresses ('ip addr del <ip> dev cilium_host')
3. Restart the agent so Reinitialize re-runs with current IPAM-allocated internal IPs
4. Ensure the agent runs with CAP_NET_ADMIN in the host netns
5. Verify the node's CiliumInternalIP from 'cilium status' matches the address being assigned

Example fix

# before: stale address conflicts
$ ip addr show cilium_host
  inet 10.0.0.99/32 scope global (stale)
# after
$ ip addr del 10.0.0.99/32 dev cilium_host
$ kubectl -n kube-system rollout restart ds/cilium

func checkCiliumHostAddrs(internalIPv4 net.IP) error {
	link, err := netlink.LinkByName("cilium_host")
	if err != nil {
		return fmt.Errorf("cilium_host missing: %w", err)
	}
	addrs, err := netlink.AddrList(link, netlink.FAMILY_V4)
	if err != nil {
		return err
	}
	for _, a := range addrs {
		if a.IP.Equal(internalIPv4) && a.IPNet.String() != internalIPv4.String()+"/32" {
			return fmt.Errorf("conflicting address %s on cilium_host; delete stale address", a.IP)
		}
	}
	return nil
}

if err := loader.Reinitialize(ctx, lnc, tc, iptMgr, p, bt); err != nil {
	if strings.Contains(err.Error(), "failed to add internal IP address to") {
		// name the device from the message and clean conflicting addrs
		return fmt.Errorf("clean stale cilium_host addresses and restart: %w", err)
	}
	return err
}

When it happens

Trigger: Reinitialize runs and addHostDeviceAddr fails adding internalIPv4 or internalIPv6 to cilium_host: netlink.AddrAdd returns an error such as permission denied, address already assigned with different scope, or the IP is invalid/duplicated on the link.

Common situations: Stale cilium_host from a prior run holding a conflicting address (RTNETLINK answers: File exists); agent missing CAP_NET_ADMIN; node IP changed after IPAM reallocation so the old internal IP conflicts; restart loops leaving the device half-configured.

Related errors

• unable to lookup route for node %s: %w
• unable to find local route for destination %s: %w
• failed to setup base devices: %w
• retrieving device %s: %w
• failed to replace IPv4 address: %w

AI-assisted analysis of cilium/cilium@ac7b90affa (2026-08-31). Data as JSON: /api/errors/012816939039f392. Report an issue: GitHub.