What causes "setting forwardable IPs for node %s: %w" in cilium/cilium?

Inspect the wrapped (%v / errors.Unwrap) inner error in the agent logs to identify the actual Set failure.

setting forwardable IPs for node %s: %w

Error message

setting forwardable IPs for node %s: %w

What it means

This error wraps a failure from ForwardableIPManager.Set when the node neighbor discovery module tries to register a node's forwardable IP addresses (used for L2 neighbor discovery) under an owner identified by the node ID. The %w wrapping preserves the underlying cause (e.g. the Set call failing to program or record the IP set). It is raised in the observer's apply step, invoked from its run job loop when node table changes are processed.

Source

Thrown at pkg/node/neighbordiscovery/node_neighbor_discovery.go:156

		txn = o.db.ReadTxn()
	}
}

func (o *nodeNeighborObserver) apply(change statedb.Change[*node.Node]) error {
	n := change.Object
	id := n.Identity()
	ips := nodeIPs(n)
	if change.Deleted || n.Local != nil {
		ips = nil
	}

	owner := neighbor.ForwardableIPOwner{
		Type: neighbor.ForwardableIPOwnerNode,
		ID:   id.String(),
	}
	if err := o.forwardableIPManager.Set(owner, maps.Keys(ips)); err != nil {
		return fmt.Errorf("setting forwardable IPs for node %s: %w", id, err)
	}
	return nil
}

func nodeIPs(n *node.Node) map[netip.Addr]struct{} {
	ips := map[netip.Addr]struct{}{}
	for _, ipv6 := range []bool{false, true} {
		if ip, ok := netip.AddrFromSlice(n.GetNodeIP(ipv6)); ok {
			ips[ip.Unmap()] = struct{}{}
		}
	}
	return ips
}

func (o *nodeNeighborObserver) NodeConfigurationChanged(config.Config) error {
	o.configInitOnce.Do(func() { close(o.configInitialized) })
	return nil
}

View on GitHub (pinned to ac7b90affa)

Solutions

Inspect the wrapped (%v / errors.Unwrap) inner error in the agent logs to identify the actual Set failure.
Verify the node's IPs (nodeIPs(n)) are valid, non-zero netip.Addr values before they reach the node table.
Check that L2 neighbor discovery is supported/functional on the host kernel; disable it if the datapath cannot program neighbor entries.
Restart the agent so the node-neighbor-discovery job re-runs apply with fresh state; check for stale neighbor state.

Example fix

// before
if err := o.forwardableIPManager.Set(owner, maps.Keys(ips)); err != nil {
    return fmt.Errorf("setting forwardable IPs for node %s: %w", id, err)
}
// after
ips := nodeIPs(n)
if len(ips) == 0 {
    o.Logger.Warn("node has no forwardable IPs, skipping", "node", id)
    return nil
}
if err := o.forwardableIPManager.Set(owner, maps.Keys(ips)); err != nil {
    return fmt.Errorf("setting forwardable IPs for node %s: %w", id, err)
}

Defensive patterns

Strategy: try-catch

Validate before calling

ips := nodeIPs(n)
if len(ips) == 0 {
    return nil // skip Set for nodes with no forwardable IPs
}

Type guard

func hasValidIPs(ips map[netip.Addr]struct{}) bool {
    for a := range ips {
        if a.IsValid() && !a.IsUnspecified() {
            return true
        }
    }
    return false
}

Try / catch

if err := o.forwardableIPManager.Set(owner, maps.Keys(ips)); err != nil {
    return fmt.Errorf("setting forwardable IPs for node %s: %w", id, err)
}
// caller: log with errors.Unwrap(err) details and continue the observer loop instead of crashing the job

Prevention

Validate node IPs before they enter the node table.
Monitor agent logs for netlink/neighbor programming failures.
Keep the host kernel up to date for neighbor-discovery support.
Test node churn (add/remove) in CI with L2 discovery enabled.

When it happens

Trigger: The statedb node table emits an upsert/change for a node; apply() constructs a ForwardableIPOwner{Type: ForwardableIPOwnerNode, ID: id.String()} and calls forwardableIPManager.Set(owner, maps.Keys(ips)), which returns a non-nil error (e.g. neighbor map programming failure in the datapath, invalid/zero IP addresses, or internal state errors).

Common situations: Nodes with no usable or malformed IPs entering the node table; kernel/netlink failures while refreshing neighbor entries; races or shutdown in progress when the manager processes updates; L2 neighbor discovery enabled in environments where the datapath cannot program forwardable IPs.

Related errors

AI-assisted analysis of cilium/cilium@ac7b90affa (2026-08-31). Data as JSON: /api/errors/c0bb3942d5c7ef85. Report an issue: GitHub.