{"record":{"id":"cced14bcec7ed713","repo":"hashicorp/nomad","slug":"stochastic-dynamic-port-selection-failed","errorCode":null,"errorMessage":"stochastic dynamic port selection failed","messagePattern":"stochastic dynamic port selection failed","errorType":"validation","errorClass":null,"httpStatus":null,"severity":"error","filePath":"nomad/structs/network.go","lineNumber":716,"sourceCode":"\n// getDynamicPortsStochastic takes the nodes used port bitmap which may be nil\n// if no ports have been allocated yet, any ports already offered in the caller,\n// and the network ask. It returns a set of unused ports to fulfil the ask's\n// DynamicPorts or an error if it failed. An error does not mean the ask can not\n// be satisfied as the method has a fixed amount of random probes and if these\n// fail, the search is aborted.\nfunc getDynamicPortsStochastic(nodeUsed Bitmap, portsInOffer []int, minDynamicPort, maxDynamicPort int, reservedPorts []Port, count int) ([]int, error) {\n\tvar reserved, dynamic []int\n\tfor _, port := range reservedPorts {\n\t\treserved = append(reserved, port.Value)\n\t}\n\n\tfor range count {\n\t\tattempts := 0\n\tPICK:\n\t\tattempts++\n\t\tif attempts > maxRandPortAttempts {\n\t\t\treturn nil, fmt.Errorf(\"stochastic dynamic port selection failed\")\n\t\t}\n\n\t\trandPort := minDynamicPort\n\t\tif maxDynamicPort-minDynamicPort > 0 {\n\t\t\trandPort = randPort + rand.Intn(maxDynamicPort-minDynamicPort)\n\t\t}\n\n\t\tif nodeUsed != nil && nodeUsed.Check(uint(randPort)) {\n\t\t\tgoto PICK\n\t\t}\n\n\t\tfor _, ports := range [][]int{reserved, dynamic} {\n\t\t\tif isPortReserved(ports, randPort) {\n\t\t\t\tgoto PICK\n\t\t\t}\n\t\t}\n\t\t// the pick conflicted with a previous pick that hasn't been saved to\n\t\t// the index yet","sourceCodeStart":698,"sourceCodeEnd":734,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/nomad/structs/network.go#L698-L734","documentation":"getDynamicPortsStochastic picks dynamic ports by repeatedly drawing random values in [minDynamicPort, maxDynamicPort) and checking collisions. After maxRandPortAttempts failed draws for a single port, it gives up with \"stochastic dynamic port selection failed\". This is the fallback/randomized path used when precise enumeration isn't possible, and it fails when the range is too full for random search to find free ports.","triggerScenarios":"AssignPorts' dynamic port ask runs maxRandPortAttempts random draws without finding a free port: the dynamic range is nearly saturated on that address (or range is tiny relative to requested count).","commonSituations":"Heavily loaded nodes with thousands of allocations; very narrow dynamic_port_range configs; tests with artificially small ranges causing >maxRandPortAttempts collisions.","solutions":["Increase the dynamic port range size on the client.","Reduce concurrent dynamic port demand (fewer dynamic ports per allocation).","Retry scheduling on another node with free port capacity.","Raise maxRandPortAttempts if ranges are legitimately dense but not exhausted (library-level tuning)."],"exampleFix":"// before\n# range too small for load\ndynamic_port_range = \"20000-20050\"\n// after\ndynamic_port_range = \"20000-40000\"","handlingStrategy":"fallback","validationCode":"utilization := float64(usedDynamicPorts(node)) / float64(rangeSize)\nif utilization > 0.9 {\n  return fmt.Errorf(\"dynamic port range %.0f%% full on %s; pick another node\", utilization*100, node.ID)\n}","typeGuard":"func rangeHealthy(used, size, attempts int) bool { return size-used > attempts }","tryCatchPattern":"null","preventionTips":["Ensure the dynamic range is orders of magnitude larger than per-node port demand","Retry scheduling with node re-selection when port selection fails","Keep dynamic_port_range within ephemeral (ip_local_port_range) limits to avoid OS-level clashes"],"tags":["nomad","ports","exhaustion","random"],"backgroundTag":"port-range-exhausted","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}