What causes "failed to remove ipset %s: %w" in cilium/cilium?

Delete iptables rules referencing the ipset first (`iptables-save | grep cilium_node_ipset` then remove them), then destroy manually.

cilium/cilium · error

failed to remove ipset %s: %w

Error message

failed to remove ipset %s: %w

What it means

Returned when `ipset destroy <name>` fails while removing a node ipset. The method first lists the set (a failed list means it doesn't exist and is a no-op success), so this error means the set exists but could not be destroyed — typically because it is still referenced by iptables rules.

Source

Thrown at pkg/datapath/iptables/ipset/ipset.go:229

	log *slog.Logger

	executable
}

func (i *ipset) create(ctx context.Context, name string, family string) error {
	if _, err := i.run(ctx, "create", name, "iphash", "family", family, "-exist"); err != nil {
		return fmt.Errorf("failed to create ipset %s: %w", name, err)
	}
	return nil
}

func (i *ipset) remove(ctx context.Context, name string) error {
	if _, err := i.run(ctx, "list", name); err != nil {
		// ipset does not exist, nothing to remove
		return nil
	}
	if _, err := i.run(ctx, "destroy", name); err != nil {
		return fmt.Errorf("failed to remove ipset %s: %w", name, err)
	}
	return nil
}

func (i *ipset) list(ctx context.Context, name string) (AddrSet, error) {
	out, err := i.run(ctx, "list", name)
	if err != nil {
		return AddrSet{}, fmt.Errorf("failed to list ipset %s: %w", name, err)
	}

	addrs := AddrSet{}
	scanner := bufio.NewScanner(bytes.NewReader(out))
	for scanner.Scan() {
		line := scanner.Text()
		addr, err := netip.ParseAddr(line)
		if err != nil {
			continue
		}

View on GitHub (pinned to ac7b90affa)

Solutions

Delete iptables rules referencing the ipset first (`iptables-save | grep cilium_node_ipset` then remove them), then destroy manually.
Restart Cilium so it removes rules before ipsets in its normal teardown order.
Ensure NET_ADMIN capability for the destroy command.
If cleanup is truly stuck, destroy manually: `ipset destroy cilium_node_ipset_v4` and inspect the kernel error.

Example fix

// before
$ ipset destroy cilium_node_ipset_v4
ipset: Set cannot be destroyed: it is in use by a kernel component

// after
$ iptables -t nat -D POSTROUTING -m set --match-set cilium_node_ipset_v4 src -j MASQUERADE
$ ipset destroy cilium_node_ipset_v4

Defensive patterns

Strategy: try-catch

Validate before calling

// ensure no iptables rules reference the set before destroying
refs, _ := exec.Command("sh", "-c", "iptables-save | grep -c -- '--match-set "+name).Output()
if n, _ := strconv.Atoi(strings.TrimSpace(string(refs))); n > 0 {
	return fmt.Errorf("ipset %s still referenced by %d iptables rules", name, n)
}

Type guard

func isIpsetInUse(err error) bool {
	return err != nil && strings.Contains(err.Error(), "in use by a kernel component")
}

Try / catch

if err := ipset.remove(ctx, name); err != nil {
	if isIpsetInUse(err) {
		removeMasqueradeRules(name) // delete referencing iptables rules
		return ipset.remove(ctx, name)
	}
	return err
}

Prevention

Tear down iptables rules referencing the set before destroying it.
Treat failed `ipset list` as already-removed (the code already does).
Run with NET_ADMIN and keep ipset installed for cleanup paths.
After disabling Cilium, grep iptables-save for cilium_node_ipset leftovers.

When it happens

Trigger: ipset.remove when the set is in use: iptables masquerade rules still reference the set (ipset refuses to destroy in-use sets), NET_ADMIN missing, or the ipset binary/module issues after list succeeded.

Common situations: Cilium teardown/disable while masquerading rules referencing CILIUM_NODE_IPSET still installed; leftover rules from a crashed previous run blocking destroy; permission problems appearing only at destroy time.

Related errors

AI-assisted analysis of cilium/cilium@ac7b90affa (2026-08-31). Data as JSON: /api/errors/beb5bf46160dc1c8. Report an issue: GitHub.