{"record":{"id":"bb4d61d72edd0a52","repo":"hashicorp/nomad","slug":"network-already-configured-but-not-found-in-state","errorCode":null,"errorMessage":"network already configured but not found in state","messagePattern":"network already configured but not found in state","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"client/allocrunner/network_hook.go","lineNumber":177,"sourceCode":"\t\t\t// recreating the netns from scratch before giving up\n\t\t\tif errors.Is(err, ErrCNICheckFailed) && !checkedOnce {\n\t\t\t\th.logger.Warn(\"network configuration check failed\", \"error\", err)\n\t\t\t\tcheckedOnce = true\n\t\t\t\tdestroyErr := h.manager.DestroyNetwork(h.alloc.ID, spec)\n\t\t\t\tif destroyErr != nil {\n\t\t\t\t\treturn fmt.Errorf(\"%w: destroying network to retry failed: %v\", err, destroyErr)\n\t\t\t\t}\n\t\t\t\tgoto CREATE\n\t\t\t}\n\n\t\t\treturn fmt.Errorf(\"failed to configure networking for alloc: %v\", err)\n\t\t}\n\t\t// A nil status indicates a netns already exists and is configured correctly.\n\t\t// It should have been saved to the local state store.\n\t\tif status == nil {\n\t\t\tstateStatus := h.networkStatus.NetworkStatus()\n\t\t\tif stateStatus == nil {\n\t\t\t\treturn errors.New(\"network already configured but not found in state\")\n\t\t\t}\n\t\t\tstatus = stateStatus\n\t\t}\n\n\t\t// If the driver set the sandbox hostname label, then we will use that\n\t\t// to set the HostsConfig.Hostname. Otherwise, identify the sandbox\n\t\t// container ID which will have been used to set the network namespace\n\t\t// hostname.\n\t\tif hostname, ok := spec.Labels[dockerNetSpecHostnameKey]; ok {\n\t\t\th.spec.HostsConfig = &drivers.HostsConfig{\n\t\t\t\tAddress:  status.Address,\n\t\t\t\tHostname: hostname,\n\t\t\t}\n\t\t} else if hostname, ok := spec.Labels[dockerNetSpecLabelKey]; ok {\n\n\t\t\t// the docker_sandbox_container_id is the full ID of the pause\n\t\t\t// container, whereas we want the shortened name that dockerd sets\n\t\t\t// as the pause container's hostname.","sourceCodeStart":159,"sourceCodeEnd":195,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/client/allocrunner/network_hook.go#L159-L195","documentation":"Raised in Prerun of the network isolation hook when the CNI status check returned nil — meaning a network namespace already exists and is configured correctly — but the hook cannot find the saved NetworkStatus in the local state store (h.networkStatus.NetworkStatus() returns nil). This indicates internal state inconsistency: the netns exists but Nomad lost the record describing it, so downstream code cannot restore the alloc's network details.","triggerScenarios":"Prerun runs; the CNI check reports the netns is already good (status == nil); the state store lookup then returns nil. Happens when the client's local state store lost the network status entry — e.g. data dir partially wiped/corrupted while the host netns persisted, or the alloc was migrated/restored from a snapshot missing the network_status object.","commonSituations":"Operator deleted or replaced the Nomad client data_dir while allocs were running; disk corruption or a bad restore of client state; upgrading Nomad where the alloc was created by an older version and state restoration skipped network status; cloning a client VM image with live alloc netns but empty state DB.","solutions":["Restart the allocation: nomad alloc stop <alloc-id> (or reschedule) so Prerun creates a fresh netns and writes a new network status to the state store.","Check the client data_dir (client.alloc_dir / state) integrity; do not delete it while allocs run — restore from backup or accept rescheduling.","If state is intentionally lost, ensure the host netns for the alloc is also cleaned up so a fresh configure path runs instead of the 'already configured' path.","Check Nomad logs around the restore for state-deserialization errors; upgrade to a version with state-restore fixes if applicable."],"exampleFix":"// before: removing client state while allocs live\nrm -rf /var/lib/nomad/client\n// after: drain then clean\nnomad node drain <node-id> -enable && nomad node drain <node-id> -disable\n# only then, if needed:\nrm -rf /var/lib/nomad/client","handlingStrategy":"fallback","validationCode":null,"typeGuard":null,"tryCatchPattern":"// on the operator side: detect and reschedule when this surfaces in client logs\nnomad alloc status <alloc-id> | grep -i 'network already configured' \\\n  && nomad alloc stop <alloc-id>","preventionTips":["Never delete/modify the Nomad client data_dir while allocations are running.","Take alloc-aware backups; don't restore client state DBs from inconsistent snapshots.","Drain nodes before VM/image cloning so netns don't survive without their state records.","Configure restart/reschedule stanzas so affected allocs are automatically recreated."],"tags":["nomad","state-store","network","netns"],"backgroundTag":"network-state-missing","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}