hashicorp/nomad · error

dynamic port selection failed

Error message

dynamic port selection failed

What it means

getDynamicPortsPrecise enumerates all free indexes in the dynamic port range (minus ports already in the current offer) via IndexesInRangeFiltered, then randomly samples numDyn of them. If fewer free ports than requested dynamic ports exist in [minDynamicPort, maxDynamicPort], it fails with "dynamic port selection failed". This means the node's dynamic port space for that address is exhausted (or over-fragmented).

Source

Thrown at nomad/structs/network.go:687

	} else {
		usedSet, err = NewBitmap(MaxValidPort)
		if err != nil {
			return nil, err
		}
	}

	for _, port := range reserved {
		usedSet.Set(uint(port.Value))
	}

	// Get the indexes of the unset ports, less those which have already been
	// picked as part of this offer
	availablePorts := usedSet.IndexesInRangeFiltered(
		false, uint(minDynamicPort), uint(maxDynamicPort), portsInOffer)

	// Randomize the amount we need
	if len(availablePorts) < numDyn {
		return nil, fmt.Errorf("dynamic port selection failed")
	}

	numAvailable := len(availablePorts)
	for i := range numDyn {
		j := rand.Intn(numAvailable)
		availablePorts[i], availablePorts[j] = availablePorts[j], availablePorts[i]
	}

	return availablePorts[:numDyn], nil
}

// getDynamicPortsStochastic takes the nodes used port bitmap which may be nil
// if no ports have been allocated yet, any ports already offered in the caller,
// and the network ask. It returns a set of unused ports to fulfil the ask's
// DynamicPorts or an error if it failed. An error does not mean the ask can not
// be satisfied as the method has a fixed amount of random probes and if these
// fail, the search is aborted.
func getDynamicPortsStochastic(nodeUsed Bitmap, portsInOffer []int, minDynamicPort, maxDynamicPort int, reservedPorts []Port, count int) ([]int, error) {

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Widen the client's dynamic port range (e.g. set dynamic_port_range = "10000-40000").
2. Reduce the number of dynamic ports the job requests.
3. Move the allocation to a less loaded node.
4. Audit for leaked/stale allocations holding ports on the node.

Example fix

// before (client.hcl)
# dynamic_port_range not set (default 20000-32000)
// after
dynamic_port_range = "10000-45000"

// compare requested dynamic port count with the node's free dynamic capacity
free := countFreePortsInRange(node, dynamicMin, dynamicMax)
if free < len(ask.DynamicPorts) {
  return fmt.Errorf("node has %d free dynamic ports, need %d", free, len(ask.DynamicPorts))
}

func hasCapacity(free, needed int) bool { return free >= needed }

null

When it happens

Trigger: AssignPorts (or its anonymous helper) requests more dynamic ports than the count of unreserved ports available in the configured dynamic range on that address, e.g. ask.DynamicPorts length > len(availablePorts).

Common situations: Node with many allocations exhausting the default 20000-32000 dynamic range; a job asking for dozens of dynamic ports; a narrowed custom dynamic range via client config.

Related errors

• stochastic dynamic port selection failed
• volume is currently unschedulable
• Missing job datacenters
• Job datacenter must be non-empty string
• Missing spread attribute

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/0e0c00ced47360c9. Report an issue: GitHub.