{"record":{"id":"d9991bb4097fa6e5","repo":"hashicorp/nomad","slug":"w-destroying-network-to-retry-failed-v","errorCode":null,"errorMessage":"%w: destroying network to retry failed: %v","messagePattern":"%w: destroying network to retry failed: (.+?)","errorType":"error_code","errorClass":null,"httpStatus":null,"severity":"error","filePath":"client/allocrunner/network_hook.go","lineNumber":165,"sourceCode":"\t}\n\n\tif spec != nil {\n\t\th.spec = spec\n\t\th.isolationSetter.SetNetworkIsolation(spec)\n\t}\n\n\tif spec != nil {\n\t\tstatus, err := h.networkConfigurator.Setup(context.TODO(), h.alloc, spec, created)\n\t\tif err != nil {\n\t\t\t// if the netns already existed but is invalid, we get\n\t\t\t// ErrCNICheckFailed. We'll try to recover from this one time by\n\t\t\t// recreating the netns from scratch before giving up\n\t\t\tif errors.Is(err, ErrCNICheckFailed) && !checkedOnce {\n\t\t\t\th.logger.Warn(\"network configuration check failed\", \"error\", err)\n\t\t\t\tcheckedOnce = true\n\t\t\t\tdestroyErr := h.manager.DestroyNetwork(h.alloc.ID, spec)\n\t\t\t\tif destroyErr != nil {\n\t\t\t\t\treturn fmt.Errorf(\"%w: destroying network to retry failed: %v\", err, destroyErr)\n\t\t\t\t}\n\t\t\t\tgoto CREATE\n\t\t\t}\n\n\t\t\treturn fmt.Errorf(\"failed to configure networking for alloc: %v\", err)\n\t\t}\n\t\t// A nil status indicates a netns already exists and is configured correctly.\n\t\t// It should have been saved to the local state store.\n\t\tif status == nil {\n\t\t\tstateStatus := h.networkStatus.NetworkStatus()\n\t\t\tif stateStatus == nil {\n\t\t\t\treturn errors.New(\"network already configured but not found in state\")\n\t\t\t}\n\t\t\tstatus = stateStatus\n\t\t}\n\n\t\t// If the driver set the sandbox hostname label, then we will use that\n\t\t// to set the HostsConfig.Hostname. Otherwise, identify the sandbox","sourceCodeStart":147,"sourceCodeEnd":183,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/client/allocrunner/network_hook.go#L147-L183","documentation":"After creating the network, the hook configures it; if configuration fails with ErrCNICheckFailed the hook retries once by destroying and recreating the netns. If DestroyNetwork also fails during that retry path, the original configuration error is wrapped with this message (using %w so errors.Is(err, ErrCNICheckFailed) still matches), aborting the allocation.","triggerScenarios":"Prerun: ConfigureNetwork returns an error that errors.Is(ErrCNICheckFailed) on first attempt, then h.manager.DestroyNetwork(h.alloc.ID, spec) returns destroyErr — the wrapped error combines both.","commonSituations":"CNI check failure due to stale/partial netns from a previous crashed allocation, plus destroy failure because the netns or veth devices are wedged, the network spec is stale/invalid, or CNI DEL plugin invocation fails (plugin missing for teardown).","solutions":["Fix the underlying CNI check failure (inner error) — typically reinstall/repair CNI plugins and validate their config.","Manually clean the wedged netns/veth on the host (ip netns / ip link) so subsequent destroy succeeds.","Restart the Nomad client to clear stale network namespace state, then reschedule the allocation.","Ensure the same CNI plugins used for ADD are available for DEL — missing plugins break DestroyNetwork.","Upgrade Nomad; several CNI retry/teardown bugs were fixed in later releases."],"exampleFix":null,"handlingStrategy":"retry","validationCode":"// before run: ensure both ADD and DEL plugin binaries exist\nfor _, p := range []string{\"bridge\", \"portmap\", \"firewall\"} {\n    if _, err := os.Stat(filepath.Join(cniPath, p)); err != nil {\n        return fmt.Errorf(\"CNI plugin %s missing; DEL will fail during retry\", p)\n    }\n}","typeGuard":"func isCNICheckFailed(err error) bool { return errors.Is(err, ErrCNICheckFailed) }","tryCatchPattern":"if err := hook.Prerun(); err != nil {\n    if errors.Is(err, ErrCNICheckFailed) {\n        // retry path also failed at destroy: clean the wedged netns/veth on the host, then reschedule\n        log.Printf(\"CNI check failed and destroy-retry failed: %v\", err)\n    }\n}","preventionTips":["Install all CNI plugin binaries used by both ADD and DEL chains.","Clean wedged netns/veth after crashed allocations (ip -n <id> link).","Restart nomad clients after host network failures.","Upgrade Nomad to pick up CNI retry/teardown fixes."],"tags":["nomad","network","cni","netns","retry"],"backgroundTag":"cni-network-configure-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}