vitessio/vitess · warning

work queue is not open

Error message

work queue is not open

What it means

The vcopier's copy work queue enqueues tasks only while it is open. enqueue is called after the queue has been closed (during shutdown/cancellation), so the task is rejected with this error to avoid scheduling copy work on a stopped vcopier.

Source

Thrown at go/vt/vttablet/tabletmanager/vreplication/vcopier.go:788

}

// close waits for all workers to be returned to the worker pool.
func (vcq *vcopierCopyWorkQueue) close() {
	if !vcq.isOpen {
		return
	}
	vcq.isOpen = false
	vcq.workerPool.Close()
}

// enqueue a new copy task. This will obtain a worker from the pool, execute
// the task with that worker, and afterwards return the worker to the pool. If
// vcopierCopyWorkQueue is configured to operate concurrently, the task will be
// executed in a separate goroutine. Otherwise the task will be executed in the
// calling goroutine.
func (vcq *vcopierCopyWorkQueue) enqueue(ctx context.Context, currT *vcopierCopyTask) error {
	if !vcq.isOpen {
		return errors.New("work queue is not open")
	}

	// Get a handle on an unused worker.
	poolH, err := vcq.workerPool.Get(ctx)
	if err != nil {
		return fmt.Errorf("failed to get a worker from pool: %s", err.Error())
	}

	currW, ok := poolH.(*vcopierCopyWorker)
	if !ok {
		return errors.New("failed to cast pool resource to *vcopierCopyWorker")
	}

	execute := func(task *vcopierCopyTask) {
		currW.execute(ctx, task)
		vcq.workerPool.Put(poolH)
	}

View on GitHub (pinned to 01a25a7d17)

Solutions

1. Check whether the workflow was cancelled or the tablet restarted mid-copy; re-run/retry the workflow — the error is expected during shutdown
2. Ensure callers stop enqueuing before closing the queue (check ordering of vcq.Close vs pending copy tasks)
3. If seen persistently outside shutdown, look for a race in vcopier teardown and report/inspect with a goroutine dump

// Check workflow/tablet state before re-running copy operations
st, _ := vtctldclient GetWorkflows --keyspace ks
// if workflow shows it was cancelled mid-copy, re-issue the workflow instead of treating as failure

if err := vcq.enqueue(ctx, task); err != nil {
    if strings.Contains(err.Error(), "work queue is not open") {
        // expected during shutdown; log and return without retry
        return nil
    }
    return err
}

When it happens

Trigger: A copy task is submitted to vcopierCopyWorkQueue.enqueue after vcq.isOpen was set false — i.e. CopyAll is shutting down (context cancel, stream ending) while a worker/goroutine still attempts to enqueue the next table copy task.

Common situations: Cancelling a MoveTables workflow mid-copy; vreplication stream ending during the copy phase; vttablet shutting down while copy tasks are in flight; race between queue close and pending enqueue calls.

Related errors

• vreplication engine is closed
• unexpected: there are no tables to copy
• CopyAll was interrupted due to context expiration
• plan not found for table: %s, current plans are: %#v
• expecting field event first, got: %v

AI-assisted analysis of vitessio/vitess@01a25a7d17 (2026-09-01). Data as JSON: /api/errors/b648ac75d5708e8f. Report an issue: GitHub.