hashicorp/nomad · error · ErrCNICheckFailed

network namespace already exists but was misconfigured

Error message

network namespace already exists but was misconfigured

What it means

ErrCNICheckFailed is a sentinel error (declared in client/allocrunner/network_hook.go) meaning Nomad's CNI post-setup verification found an existing network namespace that is present but misconfigured. It wraps the underlying CNI error in networking_cni.go:199 (fmt.Errorf("%w: %w", ErrCNICheckFailed, err)). The hook explicitly treats it as recoverable once: in network_hook.go:160 Prerun catches it via errors.Is and retries by tearing down and recreating the netns from scratch before giving up.

Source

Thrown at client/allocrunner/network_hook.go:33

	"github.com/hashicorp/nomad/plugins/drivers"
	"github.com/miekg/dns"
)

const (
	// dockerNetSpecLabelKey is the label added when we create a pause
	// container to own the network namespace, and the NetworkIsolationSpec we
	// get back from CreateNetwork has this label set as the container ID.
	// We'll use this to generate a hostname for the task in the event the user
	// did not specify a custom one. Please see dockerNetSpecHostnameKey.
	dockerNetSpecLabelKey = "docker_sandbox_container_id"

	// dockerNetSpecHostnameKey is the label added when we create a pause
	// container and the task group network include a user supplied hostname
	// parameter.
	dockerNetSpecHostnameKey = "docker_sandbox_hostname"
)

var ErrCNICheckFailed = errors.New("network namespace already exists but was misconfigured")

type networkIsolationSetter interface {
	SetNetworkIsolation(*drivers.NetworkIsolationSpec)
}

// allocNetworkIsolationSetter is a shim to allow the alloc network hook to
// set the alloc network isolation configuration without full access
// to the alloc runner
type allocNetworkIsolationSetter struct {
	ar *allocRunner
}

func (a *allocNetworkIsolationSetter) SetNetworkIsolation(n *drivers.NetworkIsolationSpec) {
	for _, tr := range a.ar.tasks {
		tr.SetNetworkIsolation(n)
	}
}

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Let Nomad self-heal: it recreates the netns once automatically; if the retry also fails, restart the allocation (nomad alloc stop <alloc-id>) so a fresh netns is created.
2. Check CNI plugin binary/config consistency in the cni_plugin_dir vs the network conflist (versions must match); re-sync plugin binaries.
3. Clean stale netns/interfaces: verify no orphaned veth/bridge interfaces or stale netns handles from crashed allocs (ip netns list, ip link), then restart nomad client.
4. Ensure nothing else mutates node networking (firewalld, kube-proxy, other CNI users); disable conflicting components.

Example fix

// HCL: pin matching CNI plugin + config versions
client {
  cni_config_dir = "/opt/cni/config"
  cni_plugin_dir = "/opt/cni/plugins" // ensure binaries here match the .conflist revision
}

// before submit, on the client node: verify CNI plugins/config are present and versions consistent
ls "$CNI_PLUGIN_DIR" && ls "$CNI_CONFIG_DIR"
# ensure no orphaned netns/veth from previous allocs
ip -n <netns> addr 2>/dev/null || true

// Nomad already retries once internally; job-level guard
job ... {
  restart { attempts = 3 interval = "10m" delay = "15s" mode = "delay" }
  reschedule { attempts = 5 interval = "1h" }
}

When it happens

Trigger: After CNI plugins run, Nomad calls the CNI CHECK command against the existing netstat. If the check command returns an error for a netns that already exists (e.g. stale pod-level netns, leftover interface config, conflicting port mappings), networking_cni.go wraps the CNI error in ErrCNICheckFailed. Surfaces through allocRunner Prerun / network hook Setup.

Common situations: Node upgrades or plugin upgrades (CNI plugin versions mismatch, e.g. conflist updated between allocs); leftover state from a crashed Nomad agent where the netns survived but is inconsistent; dangling interfaces from an unclean docker/containerd restart; bridge or iptables rules mutated by another component (firewalld reload, kube-proxy running on same node).

Related errors

• %w: destroying network to retry failed: %v
• network already configured but not found in state
• no CNI network config found
• failed to configure network manager: %v
• failed to initialize network configurator: %v

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/d4ef5e97d0fe3d5b. Report an issue: GitHub.