What causes "users: uid/gid pool exhausted" in hashicorp/nomad?

Increase the dynamic user uid/gid min/max range in client configuration to cover peak concurrent tasks

ErrPoolExhausted

ErrPoolExhausted

Error message

users: uid/gid pool exhausted

What it means

ErrPoolExhausted is returned by the dynamic uid/gid pool's Acquire when every ugID in the configured min..max range is already in use. The pool tracks allocated users in a set; when used.Size() equals the full range size, no more dynamic users can be handed out to tasks.

Source

Thrown at helper/users/dynamic/pool.go:19

// Copyright IBM Corp. 2015, 2026
// SPDX-License-Identifier: BUSL-1.1

// Package dynamic provides a way of allocating UID/GID to be used by Nomad
// tasks with no associated service users managed by the operating system.
package dynamic

import (
	"errors"
	"math/rand"
	"strconv"
	"sync"

	"github.com/hashicorp/go-set/v3"
	"github.com/hashicorp/nomad/helper"
)

var (
	ErrPoolExhausted = errors.New("users: uid/gid pool exhausted")
	ErrReleaseUnused = errors.New("users: release of unused uid/gid")
	ErrCannotParse   = errors.New("users: unable to parse uid/gid from username")
)

// none indicates no dynamic user
const none = 0

// doNotEnable indicates functionality should be disabled
const doNotEnable = -1

// A UGID is a combination User (UID) and Group (GID). Since Nomad is
// allocating these values together from the same pool it can ensure they are
// always matching values, thus encoding them with one value.
type UGID int

// String returns the string representation of a UGID.
//
// It's just the numbers.

View on GitHub (pinned to 482b49bf1a)

Solutions

Increase the dynamic user uid/gid min/max range in client configuration to cover peak concurrent tasks
Ensure tasks/allocations release their acquired ugids (Pool.Release) on shutdown
Investigate leaked allocations holding ugids and restart the Nomad client to reset the pool
Reduce per-task dynamic user usage so fewer ugids are held concurrently

Example fix

// before: pool range too small for 100 concurrent tasks
pool, _ := dynamic.NewPool(5000, 5100) // only 101 ids
// after: range sized to workload
pool, _ := dynamic.NewPool(5000, 9000) // 4001 ids

Defensive patterns

Strategy: validation

Validate before calling

if pool.Size() >= int(max-min)+1 {
  return errors.New("uid/gid pool would be exhausted; increase range or release ugids")
}

Try / catch

ugid, err := p.Acquire()
if errors.Is(err, dynamic.ErrPoolExhausted) {
  // back off / shed load / resize pool range
  return retryLater()
}

Prevention

Size the min/max ugID range to exceed peak concurrent task count
Always Release acquired ugids on allocation shutdown, including error paths
Monitor pool usage (used count vs capacity) in client metrics
Restart clients to clear leaks if exhaustion persists without matching task count

When it happens

Trigger: Calling Pool.Acquire() when the number of concurrently held dynamic users equals (p.max - p.min) + 1; i.e. all IDs in the configured range are allocated and none have been released.

Common situations: Nomad clusters running many tasks with dynamic workload users enabled where the configured uid/gid range is too small for concurrent task count; leaked allocations that never released their ugids after crashes or client restarts.

Related errors

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/97d304324de078e9. Report an issue: GitHub.