{"record":{"id":"b648ac75d5708e8f","repo":"vitessio/vitess","slug":"work-queue-is-not-open","errorCode":null,"errorMessage":"work queue is not open","messagePattern":"work queue is not open","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"go/vt/vttablet/tabletmanager/vreplication/vcopier.go","lineNumber":788,"sourceCode":"}\n\n// close waits for all workers to be returned to the worker pool.\nfunc (vcq *vcopierCopyWorkQueue) close() {\n\tif !vcq.isOpen {\n\t\treturn\n\t}\n\tvcq.isOpen = false\n\tvcq.workerPool.Close()\n}\n\n// enqueue a new copy task. This will obtain a worker from the pool, execute\n// the task with that worker, and afterwards return the worker to the pool. If\n// vcopierCopyWorkQueue is configured to operate concurrently, the task will be\n// executed in a separate goroutine. Otherwise the task will be executed in the\n// calling goroutine.\nfunc (vcq *vcopierCopyWorkQueue) enqueue(ctx context.Context, currT *vcopierCopyTask) error {\n\tif !vcq.isOpen {\n\t\treturn errors.New(\"work queue is not open\")\n\t}\n\n\t// Get a handle on an unused worker.\n\tpoolH, err := vcq.workerPool.Get(ctx)\n\tif err != nil {\n\t\treturn fmt.Errorf(\"failed to get a worker from pool: %s\", err.Error())\n\t}\n\n\tcurrW, ok := poolH.(*vcopierCopyWorker)\n\tif !ok {\n\t\treturn errors.New(\"failed to cast pool resource to *vcopierCopyWorker\")\n\t}\n\n\texecute := func(task *vcopierCopyTask) {\n\t\tcurrW.execute(ctx, task)\n\t\tvcq.workerPool.Put(poolH)\n\t}\n","sourceCodeStart":770,"sourceCodeEnd":806,"githubUrl":"https://github.com/vitessio/vitess/blob/01a25a7d176f94613b8d59d799f438380a8760e4/go/vt/vttablet/tabletmanager/vreplication/vcopier.go#L770-L806","documentation":"The vcopier's copy work queue enqueues tasks only while it is open. enqueue is called after the queue has been closed (during shutdown/cancellation), so the task is rejected with this error to avoid scheduling copy work on a stopped vcopier.","triggerScenarios":"A copy task is submitted to vcopierCopyWorkQueue.enqueue after vcq.isOpen was set false — i.e. CopyAll is shutting down (context cancel, stream ending) while a worker/goroutine still attempts to enqueue the next table copy task.","commonSituations":"Cancelling a MoveTables workflow mid-copy; vreplication stream ending during the copy phase; vttablet shutting down while copy tasks are in flight; race between queue close and pending enqueue calls.","solutions":["Check whether the workflow was cancelled or the tablet restarted mid-copy; re-run/retry the workflow — the error is expected during shutdown","Ensure callers stop enqueuing before closing the queue (check ordering of vcq.Close vs pending copy tasks)","If seen persistently outside shutdown, look for a race in vcopier teardown and report/inspect with a goroutine dump"],"exampleFix":null,"handlingStrategy":"retry","validationCode":"// Check workflow/tablet state before re-running copy operations\nst, _ := vtctldclient GetWorkflows --keyspace ks\n// if workflow shows it was cancelled mid-copy, re-issue the workflow instead of treating as failure","typeGuard":null,"tryCatchPattern":"if err := vcq.enqueue(ctx, task); err != nil {\n    if strings.Contains(err.Error(), \"work queue is not open\") {\n        // expected during shutdown; log and return without retry\n        return nil\n    }\n    return err\n}","preventionTips":["Don't manually restart copy phases while a workflow is shutting down","Check context cancellation before enqueueing follow-up work","Treat this error during cancel/PRS as expected, not a failure"],"tags":["vreplication","vcopier","shutdown","work-queue"],"backgroundTag":"queue-closed","analyzedSha":"01a25a7d176f94613b8d59d799f438380a8760e4","analyzedAt":"2026-09-01T17:28:30.605Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}