{"record":{"id":"d3322797e83b6b93","repo":"netbirdio/netbird","slug":"decrement-set-counter-w","errorCode":null,"errorMessage":"decrement set counter: %w","messagePattern":"decrement set counter: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"client/firewall/nftables/router_linux.go","lineNumber":536,"sourceCode":"\tif nftRule.Handle == 0 {\n\t\tlog.Warnf(\"route rule %s has no handle, removing stale entry\", ruleKey)\n\t\tif err := r.decrementSetCounter(nftRule); err != nil {\n\t\t\tlog.Warnf(\"decrement set counter for stale rule %s: %v\", ruleKey, err)\n\t\t}\n\t\tdelete(r.rules, ruleKey)\n\t\treturn nil\n\t}\n\n\tif err := r.deleteNftRule(nftRule, ruleKey); err != nil {\n\t\treturn fmt.Errorf(\"delete: %w\", err)\n\t}\n\n\tif err := r.conn.Flush(); err != nil {\n\t\treturn fmt.Errorf(flushError, err)\n\t}\n\n\tif err := r.decrementSetCounter(nftRule); err != nil {\n\t\treturn fmt.Errorf(\"decrement set counter: %w\", err)\n\t}\n\n\treturn nil\n}\n\nfunc (r *router) createIpSet(setName string, input setInput) (*nftables.Set, error) {\n\t// overlapping prefixes will result in an error, so we need to merge them\n\tprefixes := firewall.MergeIPRanges(input.prefixes)\n\n\tnfset := &nftables.Set{\n\t\tName:    setName,\n\t\tComment: input.set.Comment(),\n\t\tTable:   r.workTable,\n\t\t// required for prefixes\n\t\tInterval: true,\n\t\tKeyType:  r.af.setKeyType,\n\t}\n","sourceCodeStart":518,"sourceCodeEnd":554,"githubUrl":"https://github.com/netbirdio/netbird/blob/93e97f4bf1ad715072dcb3fb6cdb1763431b5a9c/client/firewall/nftables/router_linux.go#L518-L554","documentation":"Returned at router_linux.go:535 when ipsetCounter.Decrement fails after a route rule was already deleted from the kernel successfully. refcounter.Decrement (refcounter.go:167) only errors when the count drops to 1 and its remove function, deleteIpSet, fails — producing the chain \"decrement set counter: remove for key <set>: flush: <errno>\". The rule deletion itself succeeded; the damage is bookkeeping skew: the set's reference count stays elevated and the nftables set leaks in the kernel.","triggerScenarios":"The rule referenced a prefix set (expr.Lookup in the rule) whose DelSet/Flush failed: EBUSY because another rule still references the set, ENOENT because the set was already removed externally, or EPERM. Happens on the last DeleteRouteRule for a network whose set deletion races another consumer of the same hashed set.","commonSituations":"Multiple route ACL rules sharing one ipset hash; firewalld reload deleting NetBird's table out from under the refcounter; long-running agents accumulating leaked sets (visible as stale `element set` objects named by prefix hash) until restart.","solutions":["Inspect the wrapped errno: ENOENT means the set is already gone and the counter entry can be dropped; EBUSY means another rule still holds a reference and the error is transient.","Retry the DeleteRouteRule (or call RemoveAllLegacyRouteRules-equivalent cleanup) after refreshRulesMap resynchronizes rule handles; a later successful decrement self-corrects.","On EBUSY, verify with conn.GetRules that no remaining rule still contains a Lookup on that set name before forcing deletion.","As a last resort, restart the agent: Reset()/init rebuilds counters and the work table from scratch."],"exampleFix":"// before\nif err := r.decrementSetCounter(nftRule); err != nil {\n    return fmt.Errorf(\"decrement set counter: %w\", err)\n}\n\n// after\nif err := r.decrementSetCounter(nftRule); err != nil {\n    if isErrno(err, unix.ENOENT, unix.EBUSY) {\n        // rule is already deleted; counter skew self-heals on next refresh cycle\n        log.Warnf(\"decrement set counter for %s (will reconcile): %v\", ruleKey, err)\n    } else {\n        return fmt.Errorf(\"decrement set counter: %w\", err)\n    }\n}","handlingStrategy":"fallback","validationCode":"// Before decrementing, check the set is actually deletable: no other rule references it\nsets := r.findSets(nftRule)\nfor _, name := range sets {\n    if _, ok := r.ipsetCounter.Get(name); !ok {\n        continue // no counter entry; Decrement would be a no-op anyway\n    }\n}","typeGuard":"func isBenignRefcountErr(err error) bool {\n\t// ENOENT: set already gone; EBUSY: another rule still holds a reference\n\treturn isErrno(err, unix.ENOENT, unix.EBUSY)\n}","tryCatchPattern":"if err := r.decrementSetCounter(nftRule); err != nil {\n    if isBenignRefcountErr(err) {\n        // rule deletion already succeeded; skew self-heals on next refresh/removal\n        log.Warnf(\"set counter skew (will reconcile): %v\", err)\n        return nil\n    }\n    return fmt.Errorf(\"decrement set counter: %w\", err)\n}","preventionTips":["Flush rule deletions to the kernel before dropping the last set reference so deleteIpSet does not race in-flight batches.","Never decrement counters twice for the same rule; the refcount has no negative protection.","Periodically run refreshRulesMap so counter state converges with kernel reality.","Alert on monotonically growing set counts in the netbird table — the signature of a refcount leak."],"tags":["go","nftables","refcount","netlink","firewall","linux"],"backgroundTag":null,"analyzedSha":"93e97f4bf1ad715072dcb3fb6cdb1763431b5a9c","analyzedAt":"2026-08-16T03:09:19.136Z","schemaVersion":2},"datasetVersion":"2026-08-16T08:17:34.114Z"}