{"record":{"id":"aca89c6bb3d43053","repo":"vitessio/vitess","slug":"multiple-source-keyspaces-for-a-single-workflow","errorCode":null,"errorMessage":"multiple source keyspaces for a single workflow","messagePattern":"multiple source keyspaces for a single workflow","errorType":"error_code","errorClass":"ErrMultipleSourceKeyspaces","httpStatus":null,"severity":"error","filePath":"go/vt/vtctl/workflow/server.go","lineNumber":122,"sourceCode":"\tcannotSwitchFailedTabletRefresh = \"could not refresh all of the tablets involved in the operation:\\n%s\"\n\tcannotSwitchFrozen              = \"workflow is frozen\"\n\n\t// Number of LOCK TABLES cycles to perform on the sources during SwitchWrites.\n\tlockTablesCycles = 2\n\t// Time to wait between LOCK TABLES cycles on the sources during SwitchWrites.\n\tlockTablesCycleDelay = time.Duration(100 * time.Millisecond)\n\n\tSqlUnfreezeWorkflow = \"update _vt.vreplication set state='Running', message='' where db_name=%a and workflow=%a\"\n)\n\nvar (\n\t// ErrInvalidWorkflow is a catchall error type for conditions that should be\n\t// impossible when operating on a workflow.\n\tErrInvalidWorkflow = errors.New(\"invalid workflow\")\n\t// ErrMultipleSourceKeyspaces occurs when a workflow somehow has multiple\n\t// source keyspaces across different shard primaries. This should be\n\t// impossible.\n\tErrMultipleSourceKeyspaces = errors.New(\"multiple source keyspaces for a single workflow\")\n\t// ErrMultipleTargetKeyspaces occurs when a workflow somehow has multiple\n\t// target keyspaces across different shard primaries. This should be\n\t// impossible.\n\tErrMultipleTargetKeyspaces          = errors.New(\"multiple target keyspaces for a single workflow\")\n\tErrWorkflowCompleteNotFullySwitched = errors.New(\"cannot complete workflow because you have not yet switched all read and write traffic\")\n\tErrWorkflowDeleteWritesSwitched     = errors.New(\"cannot delete workflow because you have already switched write traffic\")\n)\n\n// Server provides an API to work with Vitess workflows, like vreplication\n// workflows (MoveTables, Reshard, etc) and schema migration workflows.\ntype Server struct {\n\tts  *topo.Server\n\ttmc tmclient.TabletManagerClient\n\t// Limit the number of concurrent background goroutines if needed.\n\tsem     *semaphore.Weighted\n\tenv     *vtenv.Environment\n\toptions serverOptions\n}","sourceCodeStart":104,"sourceCodeEnd":140,"githubUrl":"https://github.com/vitessio/vitess/blob/01a25a7d176f94613b8d59d799f438380a8760e4/go/vt/vtctl/workflow/server.go#L104-L140","documentation":"ErrMultipleSourceKeyspaces is returned by scanWorkflow when the vreplication streams of a single workflow report more than one source keyspace across different shard primaries. This should be impossible for a healthy workflow, so it signals inconsistent or corrupted workflow metadata.","triggerScenarios":"Calling any server API that scans a workflow (scanWorkflow) where rows in _vt.vreplication across the source shards resolve to different source keyspaces.","commonSituations":"Manual edits to _vt.vreplication, leftover rows from a merged or misconfigured Reshard, or streams created by different tool versions writing inconsistent specs.","solutions":["Inspect `select workflow, source from _vt.vreplication` on all shards to find the divergent rows.","Delete or fix the offending stream rows so all streams share one source keyspace.","Recreate the workflow if the metadata cannot be reconciled."],"exampleFix":"// before\nrows with source keyspaces: commerce, commerce2 -> scanWorkflow returns ErrMultipleSourceKeyspaces\n// after\nmysql> delete from _vt.vreplication where id=<offending id>;\n// all streams now report source keyspace 'commerce'","handlingStrategy":"type-guard","validationCode":"sources := distinctSourceKeyspaces(keyspace, workflow)\nif len(sources) > 1 { return workflow.ErrMultipleSourceKeyspaces }","typeGuard":"func isMultipleSourceKeyspaces(err error) bool {\n    return errors.Is(err, workflow.ErrMultipleSourceKeyspaces)\n}","tryCatchPattern":"if err := scanWorkflow(ctx, ks, wf); err != nil {\n    if errors.Is(err, workflow.ErrMultipleSourceKeyspaces) {\n        // repair or delete the divergent stream rows\n    }\n    return err\n}","preventionTips":["Never hand-edit _vt.vreplication source fields","After Reshard merges, verify all streams share one source keyspace","Run Workflow Show to sanity-check metadata before operations"],"tags":["vitess","vreplication","workflow","corrupt-metadata"],"backgroundTag":"invalid-workflow-state","analyzedSha":"01a25a7d176f94613b8d59d799f438380a8760e4","analyzedAt":"2026-09-01T17:28:30.605Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}