What causes "failed to get a worker from pool: %s" in vitessio/vitess?

Check if the workflow/tablet was stopping — this error is expected during cancellation; retry the copy phase

failed to get a worker from pool: %s

Error message

failed to get a worker from pool: %s

What it means

vcopier.enqueue hands each copy task to a worker from a nugget-style resource pool. If pool.Get(ctx) fails — typically because the context was cancelled while waiting for a free worker or because the pool is closed — the enqueue fails with this wrapped error.

Source

Thrown at go/vt/vttablet/tabletmanager/vreplication/vcopier.go:794

	}
	vcq.isOpen = false
	vcq.workerPool.Close()
}

// enqueue a new copy task. This will obtain a worker from the pool, execute
// the task with that worker, and afterwards return the worker to the pool. If
// vcopierCopyWorkQueue is configured to operate concurrently, the task will be
// executed in a separate goroutine. Otherwise the task will be executed in the
// calling goroutine.
func (vcq *vcopierCopyWorkQueue) enqueue(ctx context.Context, currT *vcopierCopyTask) error {
	if !vcq.isOpen {
		return errors.New("work queue is not open")
	}

	// Get a handle on an unused worker.
	poolH, err := vcq.workerPool.Get(ctx)
	if err != nil {
		return fmt.Errorf("failed to get a worker from pool: %s", err.Error())
	}

	currW, ok := poolH.(*vcopierCopyWorker)
	if !ok {
		return errors.New("failed to cast pool resource to *vcopierCopyWorker")
	}

	execute := func(task *vcopierCopyTask) {
		currW.execute(ctx, task)
		vcq.workerPool.Put(poolH)
	}

	// If the work queue is configured to work concurrently, execute the task
	// in a separate goroutine. Otherwise execute the task in the calling
	// goroutine.
	if vcq.concurrent {
		go execute(currT)
	} else {

View on GitHub (pinned to 01a25a7d17)

Solutions

Check if the workflow/tablet was stopping — this error is expected during cancellation; retry the copy phase
Increase copy worker parallelism or reduce maxDepth so tasks don't wait long for workers
Ensure ctx used for enqueue has a generous timeout
Inspect vttablet logs for pool close/shutdown events preceding the error

Defensive patterns

Strategy: try-catch

Try / catch

err := vcq.enqueue(ctx, task)
if err != nil {
  if errors.Is(err, context.Canceled) || strings.Contains(err.Error(), "failed to get a worker from pool") {
    // expected during shutdown/cancel; resubmit after recovery
  }
  return err
}

Prevention

Size the worker pool >= expected concurrent copy tasks
Don't cancel the workflow context while copies are in flight
Tune queue maxDepth relative to parallelism

When it happens

Trigger: enqueue called while all copy workers are busy and ctx is cancelled (workflow stop/delete, engine close, tablet shutdown) before a worker becomes available, or the pool has been closed.

Common situations: Workflow cancelled mid-copy; vttablet shutting down during heavy parallel copy; copy queue depth exceeding worker pool capacity combined with a short ctx timeout.

Related errors

AI-assisted analysis of vitessio/vitess@01a25a7d17 (2026-09-01). Data as JSON: /api/errors/a9cfb120e595b051. Report an issue: GitHub.