{"record":{"id":"f9593d2b303ef3d5","repo":"vitessio/vitess","slug":"vreplication-streams-are-not-frozen-on-tablet-d","errorCode":null,"errorMessage":"vreplication streams are not frozen on tablet %d","messagePattern":"vreplication streams are not frozen on tablet (.+?)","errorType":"validation","errorClass":null,"httpStatus":null,"severity":"error","filePath":"go/vt/vtctl/workflow/utils.go","lineNumber":576,"sourceCode":"\t\t\t\trec.RecordError(fmt.Errorf(\"shard %s is still serving\", source.GetShard().ShardName()))\n\t\t\t}\n\t\t\twg.Done()\n\t\t\treturn nil\n\t\t})\n\t} else {\n\t\t_ = ts.ForAllTargets(func(target *MigrationTarget) error {\n\t\t\twg.Add(1)\n\t\t\tdefer wg.Done()\n\t\t\tres, err := ts.ws.tmc.ReadVReplicationWorkflow(ctx, target.GetPrimary().Tablet, &tabletmanagerdatapb.ReadVReplicationWorkflowRequest{\n\t\t\t\tWorkflow: ts.WorkflowName(),\n\t\t\t})\n\t\t\tif err != nil {\n\t\t\t\trec.RecordError(err)\n\t\t\t\treturn nil\n\t\t\t}\n\t\t\tfor _, stream := range res.Streams {\n\t\t\t\tif stream.Message != Frozen {\n\t\t\t\t\trec.RecordError(fmt.Errorf(\"vreplication streams are not frozen on tablet %d\", target.GetPrimary().Alias.Uid))\n\t\t\t\t\treturn nil\n\t\t\t\t}\n\t\t\t}\n\t\t\treturn nil\n\t\t})\n\t}\n\twg.Wait()\n\n\tif !ts.keepRoutingRules {\n\t\t// Check if table is routable.\n\t\tif ts.MigrationType() == binlogdatapb.MigrationType_TABLES {\n\t\t\trules, err := topotools.GetRoutingRules(ctx, ts.TopoServer())\n\t\t\tif err != nil {\n\t\t\t\trec.RecordError(errors.New(\"could not get RoutingRules\"))\n\t\t\t}\n\t\t\tfor fromTable, toTables := range rules {\n\t\t\t\tfor _, toTable := range toTables {\n\t\t\t\t\tfor _, table := range ts.Tables() {","sourceCodeStart":558,"sourceCodeEnd":594,"githubUrl":"https://github.com/vitessio/vitess/blob/01a25a7d176f94613b8d59d799f438380a8760e4/go/vt/vtctl/workflow/utils.go#L558-L594","documentation":"When validating that a workflow has completed for targets of a SHARDS-type migration, this check queries vreplication streams on each target's primary tablet and expects every stream's message to equal the sentinel Frozen. Any stream whose message is not Frozen means the target has not been frozen as part of migration completion, so this error is recorded with the tablet's alias UID.","triggerScenarios":"Running validateWorkflowHasCompleted (via validate/traffic finalization) while a target primary's vreplication streams still show a running/non-frozen state (message != \"Frozen\") — streams not yet caught up and frozen, or frozen step not executed.","commonSituations":"Calling Complete/StopTraffic too early while replication is still active; a target stream stuck (not catching up) so it never reaches Frozen; target primary unreachable so the query returns stale state.","solutions":["Wait until the streams are caught up and the freeze step marks them Frozen (check `_vt.vreplication` message on the target primary), then retry validation.","If a stream is stuck, investigate replication lag/errors on the target tablet before finalizing.","Ensure the freeze/finalize step of the workflow ran on all target primaries; re-run the workflow command that performs freezing."],"exampleFix":null,"handlingStrategy":"retry","validationCode":"// confirm streams are frozen before finalizing\nrows := queryTargetPrimary(\"select message from _vt.vreplication where workflow=?\", wf)\nfor _, m := range rows {\n    if m != \"Frozen\" {\n        return fmt.Errorf(\"stream not frozen yet: %s\", m)\n    }\n}","typeGuard":null,"tryCatchPattern":"if err := validateWorkflowHasCompleted(ts); err != nil {\n    if strings.Contains(err.Error(), \"not frozen\") {\n        // streams still active: wait and retry validation\n        time.Sleep(30 * time.Second)\n        return retryValidate(ts)\n    }\n    return err\n}","preventionTips":["Wait for streams to catch up and freeze before running Complete.","Monitor _vt.vreplication message state on target primaries.","Investigate stuck/lagged streams before retrying validation."],"tags":["go","vreplication","workflow","validation"],"backgroundTag":"vreplication-streams-not-frozen","analyzedSha":"01a25a7d176f94613b8d59d799f438380a8760e4","analyzedAt":"2026-09-01T17:28:30.605Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}