hashicorp/nomad · critical

bug: pool exhausted

Error message

bug: pool exhausted

What it means

pool.Acquire returns a free UGID by first checking whether any ID is available; if it reaches the end without finding one, it panics with 'bug: pool exhausted', because exhaustion should have been detected and returned as an error earlier. Hitting this panic indicates the pool's internal bookkeeping (used set / free count) is inconsistent.

Source

Thrown at helper/users/dynamic/pool.go:148

	if p.used.Size() == int((p.max-p.min)+1) {
		return none, ErrPoolExhausted
	}

	// attempt to select a random ugid
	if id := p.random(); id != none {
		return id, nil
	}

	// slow case where we iterate each id looking for one that is not used
	for id := p.min; id <= p.max; id++ {
		if !p.used.Contains(id) {
			p.used.Insert(id)
			return id, nil
		}
	}

	// we checked for this case up top; if we get here there is a bug
	panic("bug: pool exhausted")
}

// random will attempt to select a random UGID from the pool
func (p *pool) random() UGID {
	// make up to 10 attempts to find a random unused UGID
	// if all 10 attempts fail, return the sentinel indicating as much
	const maxAttempts = 10
	size := int64(p.max - p.min)
	tries := int(min(maxAttempts, size))
	for attempt := 0; attempt < tries; attempt++ {
		id := UGID(rand.Int63n(size)) + p.min
		if p.used.Insert(id) {
			return id
		}
	}
	return none
}

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Ensure every Acquire is paired with exactly one Release; check for leaked UGIDs in task cleanup paths
2. Report as a bug if reproducing with correct Acquire/Release usage — include a goroutine trace
3. Work around by restarting the client/agent to rebuild the pool state
4. Avoid hammering Acquire on a full pool; handle the exhaustion error before this panic point is reached

defer func() { if r := recover(); r != nil { if s, ok := r.(string); ok && strings.Contains(s, "bug: pool exhausted") { log.Fatalf("UGID pool accounting bug: %v", s) }; panic(r) } }()

When it happens

Trigger: Calling Acquire concurrently (or after a failed release path) when all UGIDs in [min,max] are in the used set, and the pre-check that should return an ErrPoolExhausted-style error was bypassed due to a race or accounting bug.

Common situations: A task leaked a UGID (Acquire without a matching Release), count/used-set desynchronization under concurrency, or caller code ignoring the returned error and calling Acquire again on a full pool.

Related errors

• ErrVolumeNameExists
• ErrAllocBroadcasterClosed
• variable already holds a lock
• invalid acl policy: %v
• acquire conflict %w

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/9e7268130857a2e1. Report an issue: GitHub.