netbirdio/netbird · error

remove legacy routing rule: %w

Error message

remove legacy routing rule: %w

What it means

Returned by addLegacyRouteRule (router_linux.go:984) when the forwarding rule key firewall.GenKey(ForwardingFormat, pair) already exists and removeLegacyRouteRule fails. removeLegacyRouteRule (line 999) handles a Handle==0 stale entry gracefully (lines 1007-1014) but errors on DelRule marshal/kernel failures ("remove legacy forwarding rule ...", line 1017) or on a decrementSetCounter failure when the last set reference drops (line 1025 -> deleteIpSet flush, error 686). Legacy rules are re-added on every route update, making this replacement path hot whenever legacy mode is active.

Source

Thrown at client/firewall/nftables/router_linux.go:985

// addLegacyRouteRule adds a legacy routing rule for mgmt servers pre route acls
func (r *router) addLegacyRouteRule(pair firewall.RouterPair) error {
	sourceExp, err := r.applyNetwork(pair.Source, nil, true)
	if err != nil {
		return fmt.Errorf("apply source: %w", err)
	}

	destExp, err := r.applyNetwork(pair.Destination, nil, false)
	if err != nil {
		return fmt.Errorf("apply destination: %w", err)
	}

	exprs := buildLegacyRouteRuleExpressions(sourceExp, destExp)

	ruleKey := firewall.GenKey(firewall.ForwardingFormat, pair)

	if _, exists := r.rules[ruleKey]; exists {
		if err := r.removeLegacyRouteRule(pair); err != nil {
			return fmt.Errorf("remove legacy routing rule: %w", err)
		}
	}

	r.rules[ruleKey] = r.conn.AddRule(&nftables.Rule{
		Table:    r.workTable,
		Chain:    r.chains[chainNameRoutingFw],
		Exprs:    exprs,
		UserData: []byte(ruleKey),
	})
	return nil
}

// removeLegacyRouteRule removes a legacy routing rule for mgmt servers pre route acls
func (r *router) removeLegacyRouteRule(pair firewall.RouterPair) error {
	ruleKey := firewall.GenKey(firewall.ForwardingFormat, pair)

	rule, exists := r.rules[ruleKey]
	if !exists {

View on GitHub (pinned to 93e97f4bf1)

Solutions

1. Run refreshRulesMap before replacement so the forwarding key maps to a live handle (AddNatRule does this at line 683 — ensure the error from it was not ignored).
2. For EBUSY decrements, let the next removal cycle retry: counter entries persist and self-heal.
3. For ENOENT, treat as success — the rule is already gone from the kernel.
4. Restart the agent to rebuild map, counters, and work table when stale state keeps blocking replacement.

Example fix

// before
if _, exists := r.rules[ruleKey]; exists {
    if err := r.removeLegacyRouteRule(pair); err != nil {
        return fmt.Errorf("remove legacy routing rule: %w", err)
    }
}

// after
if _, exists := r.rules[ruleKey]; exists {
    if err := r.removeLegacyRouteRule(pair); err != nil {
        if isErrno(err, unix.ENOENT, unix.EBUSY) {
            // kernel rule gone or set still referenced; insert below still replaces it
            log.Warnf("replace legacy rule %s: %v", ruleKey, err)
            delete(r.rules, ruleKey)
        } else {
            return fmt.Errorf("remove legacy routing rule: %w", err)
        }
    }
}

// Ensure the forwarding rule has a live handle before replacement
existing, ok := r.rules[ruleKey]
if ok && existing.Handle == 0 {
    _ = r.refreshRulesMap()
    if existing, ok = r.rules[ruleKey]; ok && existing.Handle == 0 {
        delete(r.rules, ruleKey) // stale; insert below recreates it
    }
}

func hasLiveHandle(rule *nftables.Rule) bool {
	return rule != nil && rule.Handle != 0 && rule.Table != nil && rule.Chain != nil
}

func isReplaceTolerable(err error) bool {
	return isErrno(err, unix.ENOENT, unix.EBUSY)
}

if err := r.removeLegacyRouteRule(pair); err != nil {
    if isReplaceTolerable(err) {
        // rule gone (ENOENT) or set still referenced (EBUSY): insert below replaces anyway
        log.Warnf("legacy replace %s: %v", ruleKey, err)
        delete(r.rules, ruleKey)
    } else {
        return fmt.Errorf("remove legacy routing rule: %w", err)
    }
}

When it happens

Trigger: Re-applying a legacy route while the old rule's set fails to delete (EBUSY: still referenced; ENOENT: already gone; EPERM), or after a previous flush failure left a rule whose expressions cannot be marshaled by DelRule despite having a handle.

Common situations: Long-lived legacy-mode peers accumulating leaked sets so every decrement risks EBUSY; firewalld interference desynchronizing handles; RemoveNatRule hitting the same removeLegacyRouteRule (line 1482) during teardown.

Related errors

• remove prerouting rule: %w
• add legacy routing rule: %w
• add inverse nat rule: %w
• decrement set counter: %w
• error adding set %s: %w

AI-assisted analysis of netbirdio/netbird@93e97f4bf1 (2026-08-16). Data as JSON: /api/errors/02a485cbff3503d0. Report an issue: GitHub.