netbirdio/netbird · warning

remove mangle postrouting rule: %w

Error message

remove mangle postrouting rule: %w

What it means

The POSTROUTING half of cleanupDataPlaneMark(): deleting the outbound CONNMARK rule from mangle POSTROUTING failed. Like its sibling it only fires when the iptables delete command itself errors (DeleteIfExists tolerates an absent rule), the rule remains in the kernel, and the entry is kept in r.rules so a later cleanup can retry.

Source

Thrown at client/firewall/iptables/router_linux.go:527

		r.rules[markManglePost] = postRule
	}

	return nberrors.FormatErrorOrNil(merr)
}

func (r *router) cleanupDataPlaneMark() error {
	var merr *multierror.Error
	if preRule, exists := r.rules[markManglePre]; exists {
		if err := r.iptablesClient.DeleteIfExists(tableMangle, chainPREROUTING, preRule...); err != nil {
			merr = multierror.Append(merr, fmt.Errorf("remove mangle prerouting rule: %w", err))
		} else {
			delete(r.rules, markManglePre)
		}
	}

	if postRule, exists := r.rules[markManglePost]; exists {
		if err := r.iptablesClient.DeleteIfExists(tableMangle, chainPOSTROUTING, postRule...); err != nil {
			merr = multierror.Append(merr, fmt.Errorf("remove mangle postrouting rule: %w", err))
		} else {
			delete(r.rules, markManglePost)
		}
	}

	return nberrors.FormatErrorOrNil(merr)
}

func (r *router) addPostroutingRules() error {
	// First rule for outbound masquerade
	rule1 := []string{
		"-m", "mark", "--mark", fmt.Sprintf("%#x", nbnet.PreroutingFwmarkMasquerade),
		"!", "-o", "lo",
		"-j", routingFinalNatJump,
	}
	if err := r.iptablesClient.Append(tableNat, chainRTNAT, rule1...); err != nil {
		return fmt.Errorf("add outbound masquerade rule: %v", err)
	}

View on GitHub (pinned to 93e97f4bf1)

Solutions

1. List and remove leftovers manually: `sudo iptables -t mangle -D POSTROUTING -o wt0 -m conntrack --ctstate NEW -j CONNMARK --set-mark 0x...`
2. Verify the daemon still has CAP_NET_ADMIN at teardown time and that /run/xtables.lock is free
3. Read the whole multierror: paired prerouting+postrouting failures mean a host-level cause
4. Re-run `netbird down` (it is idempotent) once the host issue is cleared
5. Reboot the host as a last resort to clear stale mangle rules, then start the agent fresh

Example fix

// before
if err := r.iptablesClient.DeleteIfExists(tableMangle, chainPOSTROUTING, postRule...); err != nil {
    merr = multierror.Append(merr, fmt.Errorf("remove mangle postrouting rule: %w", err))
}

// after: retry once to ride out xtables lock contention
if err := r.iptablesClient.DeleteIfExists(tableMangle, chainPOSTROUTING, postRule...); err != nil {
    time.Sleep(200 * time.Millisecond)
    err = r.iptablesClient.DeleteIfExists(tableMangle, chainPOSTROUTING, postRule...)
    if err != nil {
        merr = multierror.Append(merr, fmt.Errorf("remove mangle postrouting rule: %w", err))
    }
}

func markRulesPresent(ipt *iptables.IPTables) (bool, error) {
    pre, err1 := ipt.List("mangle", "PREROUTING")
    post, err2 := ipt.List("mangle", "POSTROUTING")
    if err1 != nil || err2 != nil {
        return false, errors.Join(err1, err2)
    }
    for _, rs := range [][]string{pre, post} {
        for _, r := range rs {
            if strings.Contains(r, "CONNMARK") && strings.Contains(r, "--set-mark") {
                return true, nil
            }
        }
    }
    return false, nil
}

Use a bounded retry (for example two attempts with 200ms backoff) around DeleteIfExists on teardown paths; log and continue if the second attempt still fails, keeping the entry in the rules map for the next cleanup.

When it happens

Trigger: DeleteIfExists("mangle", "POSTROUTING", postRule...) failing during Reset/Stop because CAP_NET_ADMIN was lost, the iptables binary or mangle table vanished, the xtables lock is held, or another tool already rewrote the chain between the existence check and the delete.

Common situations: Teardown in containers whose netns/modules are being dismantled concurrently; `netbird down` executed after iptables packages were upgraded/removed; long-running hosts where firewalld rewrote chains; leftover marks causing policy-routing surprises after a failed teardown.

Related errors

• remove mangle prerouting rule: %w
• add mangle prerouting rule: %w
• add mangle postrouting rule: %w
• add jump to MSS clamp chain: %w
• add mangle prerouting jump rule: %v

AI-assisted analysis of netbirdio/netbird@93e97f4bf1 (2026-08-16). Data as JSON: /api/errors/881a3a240cba562b. Report an issue: GitHub.