What causes "decrement set counter: %w" in netbirdio/netbird?

Inspect the wrapped errno: ENOENT means the set is already gone and the counter entry can be dropped; EBUSY means another rule still holds a reference and the error is transient.

netbirdio/netbird · warning

decrement set counter: %w

Error message

decrement set counter: %w

What it means

Returned at router_linux.go:535 when ipsetCounter.Decrement fails after a route rule was already deleted from the kernel successfully. refcounter.Decrement (refcounter.go:167) only errors when the count drops to 1 and its remove function, deleteIpSet, fails — producing the chain "decrement set counter: remove for key <set>: flush: <errno>". The rule deletion itself succeeded; the damage is bookkeeping skew: the set's reference count stays elevated and the nftables set leaks in the kernel.

Source

Thrown at client/firewall/nftables/router_linux.go:536

	if nftRule.Handle == 0 {
		log.Warnf("route rule %s has no handle, removing stale entry", ruleKey)
		if err := r.decrementSetCounter(nftRule); err != nil {
			log.Warnf("decrement set counter for stale rule %s: %v", ruleKey, err)
		}
		delete(r.rules, ruleKey)
		return nil
	}

	if err := r.deleteNftRule(nftRule, ruleKey); err != nil {
		return fmt.Errorf("delete: %w", err)
	}

	if err := r.conn.Flush(); err != nil {
		return fmt.Errorf(flushError, err)
	}

	if err := r.decrementSetCounter(nftRule); err != nil {
		return fmt.Errorf("decrement set counter: %w", err)
	}

	return nil
}

func (r *router) createIpSet(setName string, input setInput) (*nftables.Set, error) {
	// overlapping prefixes will result in an error, so we need to merge them
	prefixes := firewall.MergeIPRanges(input.prefixes)

	nfset := &nftables.Set{
		Name:    setName,
		Comment: input.set.Comment(),
		Table:   r.workTable,
		// required for prefixes
		Interval: true,
		KeyType:  r.af.setKeyType,
	}

View on GitHub (pinned to 93e97f4bf1)

Solutions

Inspect the wrapped errno: ENOENT means the set is already gone and the counter entry can be dropped; EBUSY means another rule still holds a reference and the error is transient.
Retry the DeleteRouteRule (or call RemoveAllLegacyRouteRules-equivalent cleanup) after refreshRulesMap resynchronizes rule handles; a later successful decrement self-corrects.
On EBUSY, verify with conn.GetRules that no remaining rule still contains a Lookup on that set name before forcing deletion.
As a last resort, restart the agent: Reset()/init rebuilds counters and the work table from scratch.

Example fix

// before
if err := r.decrementSetCounter(nftRule); err != nil {
    return fmt.Errorf("decrement set counter: %w", err)
}

// after
if err := r.decrementSetCounter(nftRule); err != nil {
    if isErrno(err, unix.ENOENT, unix.EBUSY) {
        // rule is already deleted; counter skew self-heals on next refresh cycle
        log.Warnf("decrement set counter for %s (will reconcile): %v", ruleKey, err)
    } else {
        return fmt.Errorf("decrement set counter: %w", err)
    }
}

Defensive patterns

Strategy: fallback

Validate before calling

// Before decrementing, check the set is actually deletable: no other rule references it
sets := r.findSets(nftRule)
for _, name := range sets {
    if _, ok := r.ipsetCounter.Get(name); !ok {
        continue // no counter entry; Decrement would be a no-op anyway
    }
}

Type guard

func isBenignRefcountErr(err error) bool {
	// ENOENT: set already gone; EBUSY: another rule still holds a reference
	return isErrno(err, unix.ENOENT, unix.EBUSY)
}

Try / catch

if err := r.decrementSetCounter(nftRule); err != nil {
    if isBenignRefcountErr(err) {
        // rule deletion already succeeded; skew self-heals on next refresh/removal
        log.Warnf("set counter skew (will reconcile): %v", err)
        return nil
    }
    return fmt.Errorf("decrement set counter: %w", err)
}

Prevention

Flush rule deletions to the kernel before dropping the last set reference so deleteIpSet does not race in-flight batches.
Never decrement counters twice for the same rule; the refcount has no negative protection.
Periodically run refreshRulesMap so counter state converges with kernel reality.
Alert on monotonically growing set counts in the netbird table — the signature of a refcount leak.

When it happens

Trigger: The rule referenced a prefix set (expr.Lookup in the rule) whose DelSet/Flush failed: EBUSY because another rule still references the set, ENOENT because the set was already removed externally, or EPERM. Happens on the last DeleteRouteRule for a network whose set deletion races another consumer of the same hashed set.

Common situations: Multiple route ACL rules sharing one ipset hash; firewalld reload deleting NetBird's table out from under the refcounter; long-running agents accumulating leaked sets (visible as stale `element set` objects named by prefix hash) until restart.

Related errors

AI-assisted analysis of netbirdio/netbird@93e97f4bf1 (2026-08-16). Data as JSON: /api/errors/d3322797e83b6b93. Report an issue: GitHub.