{"record":{"id":"d4ef5e97d0fe3d5b","repo":"hashicorp/nomad","slug":"network-namespace-already-exists-but-was-misconfig","errorCode":null,"errorMessage":"network namespace already exists but was misconfigured","messagePattern":"network namespace already exists but was misconfigured","errorType":"exception","errorClass":"ErrCNICheckFailed","httpStatus":null,"severity":"error","filePath":"client/allocrunner/network_hook.go","lineNumber":33,"sourceCode":"\t\"github.com/hashicorp/nomad/plugins/drivers\"\n\t\"github.com/miekg/dns\"\n)\n\nconst (\n\t// dockerNetSpecLabelKey is the label added when we create a pause\n\t// container to own the network namespace, and the NetworkIsolationSpec we\n\t// get back from CreateNetwork has this label set as the container ID.\n\t// We'll use this to generate a hostname for the task in the event the user\n\t// did not specify a custom one. Please see dockerNetSpecHostnameKey.\n\tdockerNetSpecLabelKey = \"docker_sandbox_container_id\"\n\n\t// dockerNetSpecHostnameKey is the label added when we create a pause\n\t// container and the task group network include a user supplied hostname\n\t// parameter.\n\tdockerNetSpecHostnameKey = \"docker_sandbox_hostname\"\n)\n\nvar ErrCNICheckFailed = errors.New(\"network namespace already exists but was misconfigured\")\n\ntype networkIsolationSetter interface {\n\tSetNetworkIsolation(*drivers.NetworkIsolationSpec)\n}\n\n// allocNetworkIsolationSetter is a shim to allow the alloc network hook to\n// set the alloc network isolation configuration without full access\n// to the alloc runner\ntype allocNetworkIsolationSetter struct {\n\tar *allocRunner\n}\n\nfunc (a *allocNetworkIsolationSetter) SetNetworkIsolation(n *drivers.NetworkIsolationSpec) {\n\tfor _, tr := range a.ar.tasks {\n\t\ttr.SetNetworkIsolation(n)\n\t}\n}\n","sourceCodeStart":15,"sourceCodeEnd":51,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/client/allocrunner/network_hook.go#L15-L51","documentation":"ErrCNICheckFailed is a sentinel error (declared in client/allocrunner/network_hook.go) meaning Nomad's CNI post-setup verification found an existing network namespace that is present but misconfigured. It wraps the underlying CNI error in networking_cni.go:199 (fmt.Errorf(\"%w: %w\", ErrCNICheckFailed, err)). The hook explicitly treats it as recoverable once: in network_hook.go:160 Prerun catches it via errors.Is and retries by tearing down and recreating the netns from scratch before giving up.","triggerScenarios":"After CNI plugins run, Nomad calls the CNI CHECK command against the existing netstat. If the check command returns an error for a netns that already exists (e.g. stale pod-level netns, leftover interface config, conflicting port mappings), networking_cni.go wraps the CNI error in ErrCNICheckFailed. Surfaces through allocRunner Prerun / network hook Setup.","commonSituations":"Node upgrades or plugin upgrades (CNI plugin versions mismatch, e.g. conflist updated between allocs); leftover state from a crashed Nomad agent where the netns survived but is inconsistent; dangling interfaces from an unclean docker/containerd restart; bridge or iptables rules mutated by another component (firewalld reload, kube-proxy running on same node).","solutions":["Let Nomad self-heal: it recreates the netns once automatically; if the retry also fails, restart the allocation (nomad alloc stop <alloc-id>) so a fresh netns is created.","Check CNI plugin binary/config consistency in the cni_plugin_dir vs the network conflist (versions must match); re-sync plugin binaries.","Clean stale netns/interfaces: verify no orphaned veth/bridge interfaces or stale netns handles from crashed allocs (ip netns list, ip link), then restart nomad client.","Ensure nothing else mutates node networking (firewalld, kube-proxy, other CNI users); disable conflicting components."],"exampleFix":"// HCL: pin matching CNI plugin + config versions\nclient {\n  cni_config_dir = \"/opt/cni/config\"\n  cni_plugin_dir = \"/opt/cni/plugins\" // ensure binaries here match the .conflist revision\n}","handlingStrategy":"retry","validationCode":"// before submit, on the client node: verify CNI plugins/config are present and versions consistent\nls \"$CNI_PLUGIN_DIR\" && ls \"$CNI_CONFIG_DIR\"\n# ensure no orphaned netns/veth from previous allocs\nip -n <netns> addr 2>/dev/null || true","typeGuard":null,"tryCatchPattern":"// Nomad already retries once internally; job-level guard\njob ... {\n  restart { attempts = 3 interval = \"10m\" delay = \"15s\" mode = \"delay\" }\n  reschedule { attempts = 5 interval = \"1h\" }\n}","preventionTips":["Keep cni_plugin_dir binaries and cni_config_dir conflate files from the same release.","Do not run conflicting networking controllers (kube-proxy/firewalld) on Nomad clients.","Drain nodes before client data_dir maintenance so netns and state are cleaned together.","Monitor for ErrCNICheckFailed warnings and treat repeats as node-health signals."],"tags":["cni","network","nomad","netns"],"backgroundTag":"cni-check-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}