netbirdio/netbird · error

error while adding marking rule for %s: %v

Error message

error while adding marking rule for %s: %v

What it means

addNatRule() inserts the MARK rule built for the RouterPair at position 1 of NETBIRD-RT-PRE (mangle table), stamping NEW connections with PreroutingFwmarkMasquerade or ...MasqueradeReturn so the later postrouting rules masquerade them. Failure aborts the pair add; note the source comment admits the ipset refcount incremented earlier is NOT rolled back (TODO), so failed adds can leak ipset references.

Source

Thrown at client/firewall/iptables/router_linux.go:706

	if err != nil {
		return fmt.Errorf("apply network -s: %w", err)
	}
	destExp, err := r.applyNetwork("-d", pair.Destination, nil)
	if err != nil {
		return fmt.Errorf("apply network -d: %w", err)
	}

	rule = append(rule, sourceExp...)
	rule = append(rule, destExp...)
	rule = append(rule,
		"-j", "MARK", "--set-mark", fmt.Sprintf("%#x", markValue),
	)

	// Ensure nat rules come first, so the mark can be overwritten.
	// Currently overwritten by the dst-type LOCAL rules for redirected traffic.
	if err := r.iptablesClient.Insert(tableMangle, chainRTPRE, 1, rule...); err != nil {
		// TODO: rollback ipset counter
		return fmt.Errorf("error while adding marking rule for %s: %v", pair.Destination, err)
	}

	r.rules[ruleKey] = rule

	r.updateState()
	return nil
}

func (r *router) removeNatRule(pair firewall.RouterPair) error {
	ruleKey := firewall.GenKey(firewall.NatFormat, pair)

	if rule, exists := r.rules[ruleKey]; exists {
		if err := r.iptablesClient.DeleteIfExists(tableMangle, chainRTPRE, rule...); err != nil {
			return fmt.Errorf("error while removing marking rule for %s: %v", pair.Destination, err)
		}
		delete(r.rules, ruleKey)

		if err := r.decrementSetCounter(rule); err != nil {

View on GitHub (pinned to 93e97f4bf1)

Solutions

1. Reproduce with the rule spec from the log: `sudo iptables -t mangle -I NETBIRD-RT-PRE 1 ...`
2. `modprobe xt_mark iptable_mangle` (plus ip6 variants for v6 pairs)
3. Confirm NETBIRD-RT-PRE exists; if the router was never initialized, look for the earlier createContainers error instead
4. Ensure no process holds /run/xtables.lock during route updates
5. After repeated failures, check for orphaned ipsets (`ipset list -n | grep -i nb`) and report the refcount leak upstream

Example fix

// before: ipset refcount incremented in applyNetwork is never rolled back
if err := r.iptablesClient.Insert(tableMangle, chainRTPRE, 1, rule...); err != nil {
    // TODO: rollback ipset counter
    return fmt.Errorf("error while adding marking rule for %s: %v", pair.Destination, err)
}

// after: decrement the refcount so failed adds do not leak ipsets
if err := r.iptablesClient.Insert(tableMangle, chainRTPRE, 1, rule...); err != nil {
    if derr := r.decrementSetCounter(rule); derr != nil {
        log.Warnf("rollback ipset refcount: %v", derr)
    }
    return fmt.Errorf("add marking rule for %s: %w", pair.Destination, err)
}

func markTargetSupported(ipt *iptables.IPTables) error {
    if err := ipt.NewChain("mangle", "NB-PROBE"); err != nil {
        return err
    }
    defer ipt.ClearAndDeleteChain("mangle", "NB-PROBE")
    return ipt.Append("mangle", "NB-PROBE", "-j", "MARK", "--set-mark", "0x1")
}

Catch in the route manager: roll back the ipset refcount taken during applyNetwork, keep prior rules intact, and surface the pair destination so operators can retry that single route.

When it happens

Trigger: `iptables -t mangle -I NETBIRD-RT-PRE 1 <rule>` failing: xt_mark target unavailable, mangle table inaccessible, CAP_NET_ADMIN missing, xtables lock held, or the chain NETBIRD-RT-PRE vanished because init/createContainers never ran or was externally undone. Rule contents (mark match, conntrack, ipset match) are appended before the insert, so an invalid ipset name also surfaces here.

Common situations: Route update storms from management hitting xtables lock contention; agents on hosts where xt_mark is a module that was never loaded; containers running an embedded client (client/embed) without full NET_ADMIN; leftover refcounts accumulating after repeated failures (the TODO in the source).

Related errors

• add mangle prerouting rule: %w
• add mangle postrouting rule: %w
• remove mangle prerouting rule: %w
• remove mangle postrouting rule: %w
• add jump to MSS clamp chain: %w

AI-assisted analysis of netbirdio/netbird@93e97f4bf1 (2026-08-16). Data as JSON: /api/errors/e680ad0d0d3c882e. Report an issue: GitHub.