hashicorp/nomad · error

ErrCNICheckFailed

ErrCNICheckFailed

Error message

%w: %w

What it means

When an allocation's network namespace already exists, Setup runs CNI CHECK to validate existing networking (requires bridge plugin >=1.3.0). If cni.Check fails, Nomad wraps it with the sentinel ErrCNICheckFailed so callers can detect a broken/missing netns setup distinctly from a fresh-setup failure.

Source

Thrown at client/allocrunner/networking_cni.go:199

		cniArgs[ConsulIPTablesConfigEnvVar] = string(iptablesCfg)
	}

	if !created {
		// The netns will not be created if it already exists, typically on
		// agent restart. If the configuration of a prexisting netns is wrong
		// (ex. after a host reboot for docker created netns), networking will
		// be broken. CNI's ADD command is not idempotent so we can't simply try
		// again. Run CHECK to verify the network is still valid. Older plugins
		// have a broken CHECK, so we have to allow the buggy behavior in the
		// case of a host reboot with docker-created netns there.
		cniVersion, err := version.NewSemver(c.nodeAttrs["plugins.cni.version.bridge"])
		if err == nil && supportsCNICheck.Check(cniVersion) {
			err := c.cni.Check(ctx, alloc.ID, spec.Path,
				c.nsOpts.withCapabilityPortMap(portMaps.ports),
				c.nsOpts.withArgs(cniArgs),
			)
			if err != nil {
				return nil, fmt.Errorf("%w: %w", ErrCNICheckFailed, err)
			}
		} else {
			c.logger.Debug("network namespace exists but could not check if networking is valid because bridge plugin version was <1.3.0: continuing anyways")
			return nil, nil
		}
		c.logger.Trace("network namespace exists and passed check: skipping setup")
		return nil, nil
	}

	// Depending on the version of bridge cni plugin used, a known race could occure
	// where two alloc attempt to create the nomad bridge at the same time, resulting
	// in one of them to fail. This rety attempts to overcome those erroneous failures.
	const retry = 3
	var firstError error
	var res *cni.Result
	for attempt := 1; ; attempt++ {
		var err error
		if res, err = c.cni.Setup(ctx, alloc.ID, spec.Path,

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Let Nomad tear down and re-create: stop the allocation (nomad alloc stop) or restart the client so a fresh CNI ADD runs.
2. Inspect the wrapped error from cni.Check for the failing plugin and fix host networking state (recreate bridge, restore iptables).
3. Match CNI plugin versions on the host with the version reported by the CNI conf (>=1.3.0 required for CHECK).
4. Handle ErrCNICheckFailed in custom code with errors.Is to distinguish from other setup failures.

Example fix

// before
if err := c.cni.Check(...); err != nil { return nil, err }
// after
if err := c.cni.Check(...); err != nil {
  if errors.Is(err, ErrCNICheckFailed) {
    // recreate netns via full teardown/setup
  }
  return nil, fmt.Errorf("%w: %w", ErrCNICheckFailed, err)
}

// detect stale netns before setup
if _, err := os.Stat(spec.Path); err != nil { /* netns missing; force fresh ADD */ }

func isCNICheckFailed(err error) bool {
  return errors.Is(err, ErrCNICheckFailed)
}

status, err := cfg.Setup(ctx, alloc, spec, created)
if isCNICheckFailed(err) {
  // teardown netns and retry Setup once for a clean CNI ADD
  _ = cfg.Teardown(ctx, alloc, spec, false)
  status, err = cfg.Setup(ctx, alloc, spec, false)
}

When it happens

Trigger: Alloc netns exists, the CNI config reports support for CHECK (bridge plugin version >=1.3.0), and c.cni.Check(...) returns an error — e.g. netns missing/corrupt, interface gone, iptables chain missing, or plugin binary changed since ADD.

Common situations: Client restart with stale netns after host reboot wiped iptables/links; operator deleted the bridge or veth manually; CNI plugin upgrade changing behavior; disk/netns cleanup utilities removing /var/run/netns entries.

Related errors

• network namespace already exists but was misconfigured
• no CNI network config found
• DriverStatsNotImplemented
• Error listing host volumes
• failed to configure network manager: %v

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/38f816542709e18b. Report an issue: GitHub.