netbirdio/netbird · warning

remove mangle prerouting rule: %w

Error message

remove mangle prerouting rule: %w

What it means

Raised by cleanupDataPlaneMark() when removing the previously added CONNMARK rule from mangle PREROUTING during agent teardown or reset. go-iptables DeleteIfExists() only errors when the iptables invocation itself fails (a missing rule is silently tolerated), so this means the delete could not be executed, and the stale fwmark rule stays in the kernel while the in-memory bookkeeping keeps the entry for a later retry.

Source

Thrown at client/firewall/iptables/router_linux.go:519

		"-o", r.wgIface.Name(),
		"-m", "conntrack", "--ctstate", "NEW",
		"-j", "CONNMARK", "--set-mark", fmt.Sprintf("%#x", nbnet.DataPlaneMarkOut),
	}

	if err := r.iptablesClient.AppendUnique(tableMangle, chainPOSTROUTING, postRule...); err != nil {
		merr = multierror.Append(merr, fmt.Errorf("add mangle postrouting rule: %w", err))
	} else {
		r.rules[markManglePost] = postRule
	}

	return nberrors.FormatErrorOrNil(merr)
}

func (r *router) cleanupDataPlaneMark() error {
	var merr *multierror.Error
	if preRule, exists := r.rules[markManglePre]; exists {
		if err := r.iptablesClient.DeleteIfExists(tableMangle, chainPREROUTING, preRule...); err != nil {
			merr = multierror.Append(merr, fmt.Errorf("remove mangle prerouting rule: %w", err))
		} else {
			delete(r.rules, markManglePre)
		}
	}

	if postRule, exists := r.rules[markManglePost]; exists {
		if err := r.iptablesClient.DeleteIfExists(tableMangle, chainPOSTROUTING, postRule...); err != nil {
			merr = multierror.Append(merr, fmt.Errorf("remove mangle postrouting rule: %w", err))
		} else {
			delete(r.rules, markManglePost)
		}
	}

	return nberrors.FormatErrorOrNil(merr)
}

func (r *router) addPostroutingRules() error {
	// First rule for outbound masquerade

View on GitHub (pinned to 93e97f4bf1)

Solutions

1. Inspect the returned multierror: both prerouting and postrouting failing points to a host-level iptables problem, not a stale rule
2. Run `sudo iptables -t mangle -S PREROUTING | grep CONNMARK` and delete leftovers by hand: `sudo iptables -t mangle -D PREROUTING -i wt0 -m conntrack --ctstate NEW -j CONNMARK --set-mark 0x...`
3. Ensure `netbird down` runs as root so the daemon retains CAP_NET_ADMIN
4. Check for /run/xtables.lock holders (lsof /run/xtables.lock) and stop them
5. After manual cleanup, restart the agent so its state map resyncs with the kernel

Example fix

// before: on failure the entry stays, but callers often ignore the multierror
if err := r.iptablesClient.DeleteIfExists(tableMangle, chainPREROUTING, preRule...); err != nil {
    merr = multierror.Append(merr, fmt.Errorf("remove mangle prerouting rule: %w", err))
}

// after: fall back to a position-based delete when the stored spec no longer matches
if err := r.iptablesClient.DeleteIfExists(tableMangle, chainPREROUTING, preRule...); err != nil {
    log.Warnf("rule delete failed, listing chain for manual match: %v", err)
    merr = multierror.Append(merr, fmt.Errorf("remove mangle prerouting rule: %w", err))
}

func staleConnmarkRules(ipt *iptables.IPTables, iface string) [][]string {
    rules, err := ipt.List("mangle", "PREROUTING")
    if err != nil {
        return nil
    }
    var stale [][]string
    for _, r := range rules {
        if strings.Contains(r, iface) && strings.Contains(r, "CONNMARK") {
            stale = append(stale, []string{r})
        }
    }
    return stale
}

Collect teardown errors with multierror and never abort the remaining cleanup steps; retry the whole cleanup once after a short backoff to ride out xtables lock contention before surfacing the error.

When it happens

Trigger: router.Reset()/Stop() -> cleanupDataPlaneMark() -> DeleteIfExists("mangle", "PREROUTING", preRule...) failing due to lost CAP_NET_ADMIN, iptables binary removed since setup, xtables lock held, or the mangle table becoming unavailable (module unload). Accumulated into a multierror together with the postrouting variant.

Common situations: Running `netbird down` from an unprivileged shell while the daemon already dropped privileges; container being torn down with modules unloaded first; host where another admin flushed tables concurrently so Exists-then-Delete races; agent crash recovery on next start hitting leftover rules.

Related errors

• remove mangle postrouting rule: %w
• add mangle prerouting rule: %w
• add mangle postrouting rule: %w
• add jump to MSS clamp chain: %w
• add mangle prerouting jump rule: %v

AI-assisted analysis of netbirdio/netbird@93e97f4bf1 (2026-08-16). Data as JSON: /api/errors/6b7c6092858d6c87. Report an issue: GitHub.