hashicorp/nomad · error

%w: destroying network to retry failed: %v

Error message

%w: destroying network to retry failed: %v

What it means

After creating the network, the hook configures it; if configuration fails with ErrCNICheckFailed the hook retries once by destroying and recreating the netns. If DestroyNetwork also fails during that retry path, the original configuration error is wrapped with this message (using %w so errors.Is(err, ErrCNICheckFailed) still matches), aborting the allocation.

Source

Thrown at client/allocrunner/network_hook.go:165

	}

	if spec != nil {
		h.spec = spec
		h.isolationSetter.SetNetworkIsolation(spec)
	}

	if spec != nil {
		status, err := h.networkConfigurator.Setup(context.TODO(), h.alloc, spec, created)
		if err != nil {
			// if the netns already existed but is invalid, we get
			// ErrCNICheckFailed. We'll try to recover from this one time by
			// recreating the netns from scratch before giving up
			if errors.Is(err, ErrCNICheckFailed) && !checkedOnce {
				h.logger.Warn("network configuration check failed", "error", err)
				checkedOnce = true
				destroyErr := h.manager.DestroyNetwork(h.alloc.ID, spec)
				if destroyErr != nil {
					return fmt.Errorf("%w: destroying network to retry failed: %v", err, destroyErr)
				}
				goto CREATE
			}

			return fmt.Errorf("failed to configure networking for alloc: %v", err)
		}
		// A nil status indicates a netns already exists and is configured correctly.
		// It should have been saved to the local state store.
		if status == nil {
			stateStatus := h.networkStatus.NetworkStatus()
			if stateStatus == nil {
				return errors.New("network already configured but not found in state")
			}
			status = stateStatus
		}

		// If the driver set the sandbox hostname label, then we will use that
		// to set the HostsConfig.Hostname. Otherwise, identify the sandbox

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Fix the underlying CNI check failure (inner error) — typically reinstall/repair CNI plugins and validate their config.
2. Manually clean the wedged netns/veth on the host (ip netns / ip link) so subsequent destroy succeeds.
3. Restart the Nomad client to clear stale network namespace state, then reschedule the allocation.
4. Ensure the same CNI plugins used for ADD are available for DEL — missing plugins break DestroyNetwork.
5. Upgrade Nomad; several CNI retry/teardown bugs were fixed in later releases.

// before run: ensure both ADD and DEL plugin binaries exist
for _, p := range []string{"bridge", "portmap", "firewall"} {
    if _, err := os.Stat(filepath.Join(cniPath, p)); err != nil {
        return fmt.Errorf("CNI plugin %s missing; DEL will fail during retry", p)
    }
}

func isCNICheckFailed(err error) bool { return errors.Is(err, ErrCNICheckFailed) }

if err := hook.Prerun(); err != nil {
    if errors.Is(err, ErrCNICheckFailed) {
        // retry path also failed at destroy: clean the wedged netns/veth on the host, then reschedule
        log.Printf("CNI check failed and destroy-retry failed: %v", err)
    }
}

When it happens

Trigger: Prerun: ConfigureNetwork returns an error that errors.Is(ErrCNICheckFailed) on first attempt, then h.manager.DestroyNetwork(h.alloc.ID, spec) returns destroyErr — the wrapped error combines both.

Common situations: CNI check failure due to stale/partial netns from a previous crashed allocation, plus destroy failure because the netns or veth devices are wedged, the network spec is stale/invalid, or CNI DEL plugin invocation fails (plugin missing for teardown).

Related errors

• network namespace already exists but was misconfigured
• network already configured but not found in state
• no CNI network config found
• failed to configure network manager: %v
• failed to initialize network configurator: %v

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/d9991bb4097fa6e5. Report an issue: GitHub.