hashicorp/nomad · error

network already configured but not found in state

Error message

network already configured but not found in state

What it means

Raised in Prerun of the network isolation hook when the CNI status check returned nil — meaning a network namespace already exists and is configured correctly — but the hook cannot find the saved NetworkStatus in the local state store (h.networkStatus.NetworkStatus() returns nil). This indicates internal state inconsistency: the netns exists but Nomad lost the record describing it, so downstream code cannot restore the alloc's network details.

Source

Thrown at client/allocrunner/network_hook.go:177

			// recreating the netns from scratch before giving up
			if errors.Is(err, ErrCNICheckFailed) && !checkedOnce {
				h.logger.Warn("network configuration check failed", "error", err)
				checkedOnce = true
				destroyErr := h.manager.DestroyNetwork(h.alloc.ID, spec)
				if destroyErr != nil {
					return fmt.Errorf("%w: destroying network to retry failed: %v", err, destroyErr)
				}
				goto CREATE
			}

			return fmt.Errorf("failed to configure networking for alloc: %v", err)
		}
		// A nil status indicates a netns already exists and is configured correctly.
		// It should have been saved to the local state store.
		if status == nil {
			stateStatus := h.networkStatus.NetworkStatus()
			if stateStatus == nil {
				return errors.New("network already configured but not found in state")
			}
			status = stateStatus
		}

		// If the driver set the sandbox hostname label, then we will use that
		// to set the HostsConfig.Hostname. Otherwise, identify the sandbox
		// container ID which will have been used to set the network namespace
		// hostname.
		if hostname, ok := spec.Labels[dockerNetSpecHostnameKey]; ok {
			h.spec.HostsConfig = &drivers.HostsConfig{
				Address:  status.Address,
				Hostname: hostname,
			}
		} else if hostname, ok := spec.Labels[dockerNetSpecLabelKey]; ok {

			// the docker_sandbox_container_id is the full ID of the pause
			// container, whereas we want the shortened name that dockerd sets
			// as the pause container's hostname.

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Restart the allocation: nomad alloc stop <alloc-id> (or reschedule) so Prerun creates a fresh netns and writes a new network status to the state store.
2. Check the client data_dir (client.alloc_dir / state) integrity; do not delete it while allocs run — restore from backup or accept rescheduling.
3. If state is intentionally lost, ensure the host netns for the alloc is also cleaned up so a fresh configure path runs instead of the 'already configured' path.
4. Check Nomad logs around the restore for state-deserialization errors; upgrade to a version with state-restore fixes if applicable.

Example fix

// before: removing client state while allocs live
rm -rf /var/lib/nomad/client
// after: drain then clean
nomad node drain <node-id> -enable && nomad node drain <node-id> -disable
# only then, if needed:
rm -rf /var/lib/nomad/client

// on the operator side: detect and reschedule when this surfaces in client logs
nomad alloc status <alloc-id> | grep -i 'network already configured' \
  && nomad alloc stop <alloc-id>

When it happens

Trigger: Prerun runs; the CNI check reports the netns is already good (status == nil); the state store lookup then returns nil. Happens when the client's local state store lost the network status entry — e.g. data dir partially wiped/corrupted while the host netns persisted, or the alloc was migrated/restored from a snapshot missing the network_status object.

Common situations: Operator deleted or replaced the Nomad client data_dir while allocs were running; disk corruption or a bad restore of client state; upgrading Nomad where the alloc was created by an older version and state restoration skipped network status; cloning a client VM image with live alloc netns but empty state DB.

Related errors

• network namespace already exists but was misconfigured
• %w: destroying network to retry failed: %v
• error parsing: root should be an object
• cannot specify Accessor ID
• no CNI network config found

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/bb4d61d72edd0a52. Report an issue: GitHub.