{"record":{"id":"95e3a0805f40a3cf","repo":"cilium/cilium","slug":"failed-to-delete-direct-route-q-w","errorCode":null,"errorMessage":"failed to delete direct route %q: %w","messagePattern":"failed to delete direct route %q: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"pkg/datapath/linux/node.go","lineNumber":341,"sourceCode":"\t\tDst:      netipx.PrefixIPNet(prefix),\n\t\tGw:       nodeIP,\n\t\tProtocol: linux_defaults.RTProto,\n\t}\n\n\troutes, err := safenetlink.RouteListFiltered(family, filter, netlink.RT_FILTER_DST|netlink.RT_FILTER_GW)\n\tif err != nil {\n\t\tn.log.Error(\"Unable to list direct routes\", logfields.Error, err)\n\t\treturn fmt.Errorf(\"failed to list direct routes %s: %w\", familyStr, err)\n\t}\n\n\tvar errs error\n\tfor _, rt := range routes {\n\t\tif err := netlink.RouteDel(&rt); err != nil {\n\t\t\tn.log.Warn(\"Unable to delete direct node route\",\n\t\t\t\tlogfields.CIDR, rt,\n\t\t\t\tlogfields.Error, err,\n\t\t\t)\n\t\t\terrs = errors.Join(errs, fmt.Errorf(\"failed to delete direct route %q: %w\", rt.String(), err))\n\t\t}\n\t}\n\treturn errs\n}\n\n// createNodeRouteSpec creates a route spec that points the specified prefix to the host\n// device via the router IP. The route is configured with a computed MTU for non-local\n// nodes (i.e isLocalNode is set to false).\n//\n// Example:\n// 10.10.0.0/24 via 10.10.0.1 dev cilium_host src 10.10.0.1\n// f00d::a0a:0:0:0/112 via f00d::a0a:0:0:1 dev cilium_host src fd04::11 metric 1024 pref medium\nfunc (n *linuxNodeHandler) createNodeRouteSpec(prefix netip.Prefix, isLocalNode bool) (route.Route, error) {\n\tvar (\n\t\tlocal   net.IP\n\t\tnexthop *net.IP\n\t\tmtu     int\n\t)","sourceCodeStart":323,"sourceCodeEnd":359,"githubUrl":"https://github.com/cilium/cilium/blob/ac7b90affa4baf0642e6685319d56907b3a73a6d/pkg/datapath/linux/node.go#L323-L359","documentation":"Thrown in deleteDirectRoute when netlink.RouteDel fails to remove one specific direct node route after it was successfully listed. Individual failures are logged as warnings and aggregated via errors.Join, so a single bad route does not stop deletion of the others; the joined aggregate is wrapped by the caller. This leaves the failing route installed on the host.","triggerScenarios":"During deleteAllDirectRoutes (nodeUpdate removed CIDRs) or nodeDelete, netlink.RouteDel returns an error for a listed route: typically EPERM (missing capability), EINVAL (route already changed/removed by another writer), or ENODEV (the route's link index disappeared between listing and deletion).","commonSituations":"Races with the kernel removing the interface or with other route managers (e.g. static NetworkManager routes, other CNI remnants) deleting or mutating the same route; unprivileged execution; stale routes referencing a deleted device after node restart.","solutions":["Re-run node update / restart cilium-agent: this error is often transient (route vanished mid-scan) and the next reconcile clears it.","Ensure CAP_NET_ADMIN is granted to the agent process for RouteDel.","Check for conflicting route managers (NetworkManager, dhclient, other CNIs) mutating the same routes and disable the conflict.","Use `ip route del` manually for the specific %q route printed in the error, then verify agent reconciliation succeeds."],"exampleFix":"// before: treating the aggregate as fatal and crashing\nif err := n.deleteAllDirectRoutes(removedCIDRs, oldIP); err != nil {\n    return fmt.Errorf(\"failed to delete all direct routes: %w\", err)\n}\n// after: log and continue, letting periodic reconciliation retry\nif err := n.deleteAllDirectRoutes(removedCIDRs, oldIP); err != nil {\n    n.log.Warn(\"some direct routes could not be deleted, will retry on next reconcile\",\n        logfields.Error, err)\n}","handlingStrategy":"retry","validationCode":"// preflight: confirm the route still exists right before deleting it\nroutes, err := safenetlink.RouteListFiltered(family, filter, netlink.RT_FILTER_DST|netlink.RT_FILTER_GW)\nif err != nil || len(routes) == 0 {\n    return nil // nothing to delete; avoids ESRCH/EINVAL on stale routes\n}","typeGuard":null,"tryCatchPattern":"if err := netlink.RouteDel(&rt); err != nil {\n    var errno syscall.Errno\n    if errors.As(err, &errno) && (errno == syscall.ESRCH || errno == syscall.ENODEV) {\n        // route already gone or device removed — treat as success, continue\n        continue\n    }\n    errs = errors.Join(errs, err) // real failure: aggregate and retry later\n}","preventionTips":["Treat ESRCH/ENODEV as already-deleted and continue instead of failing.","Disable conflicting route managers (NetworkManager, other CNIs) on Cilium-managed tables.","Run with CAP_NET_ADMIN so deletion is not rejected outright.","Periodically reconcile node routes so partially failed deletions are retried."],"tags":["netlink","routing","linux","race-condition"],"backgroundTag":"netlink-route-deletion-failed","analyzedSha":"ac7b90affa4baf0642e6685319d56907b3a73a6d","analyzedAt":"2026-08-31T18:27:15.868Z","schemaVersion":2},"datasetVersion":"2026-08-31T19:17:28.585Z"}