{"record":{"id":"5643c8796e678696","repo":"vitessio/vitess","slug":"failed-to-delete-post-copy-action-for-the-q-table","errorCode":null,"errorMessage":"failed to delete post copy action for the %q table with id %d: %v","messagePattern":"failed to delete post copy action for the %q table with id (.+?): (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"go/vt/vttablet/tabletmanager/vreplication/vreplicator.go","lineNumber":993,"sourceCode":"\t\treturn err\n\t}\n\t// qr should never be nil, but check anyway to be extra safe.\n\tif idqr == nil || len(idqr.Rows) != 1 {\n\t\treturn fmt.Errorf(\"unexpected number of rows returned (%d) from connection_id() query\", len(idqr.Rows))\n\t}\n\tconnID, err := idqr.Rows[0][0].ToInt64()\n\tif err != nil || connID == 0 {\n\t\treturn fmt.Errorf(\"unexpected result (%d) from connection_id() query, error: %v\", connID, err)\n\t}\n\n\tdeleteAction := func(dbc *vdbClient, id int64, vid int32, tn string) error {\n\t\tdelq, err := sqlparser.ParseAndBind(sqlDeletePostCopyAction, sqltypes.Int32BindVariable(vid),\n\t\t\tsqltypes.StringBindVariable(tn), sqltypes.Int64BindVariable(id))\n\t\tif err != nil {\n\t\t\treturn err\n\t\t}\n\t\tif _, err := dbc.ExecuteFetch(delq, 1); err != nil {\n\t\t\treturn fmt.Errorf(\"failed to delete post copy action for the %q table with id %d: %v\",\n\t\t\t\ttableName, id, err)\n\t\t}\n\t\treturn nil\n\t}\n\n\t// This could take hours so we start a monitoring goroutine to\n\t// listen for the cancellations which indicate that the controller\n\t// is stopping: engine shutdown (tablet shutdown or transition) or\n\t// controller stop (the workflow is being stopped, deleted, or\n\t// updated). If either happens we KILL the connection being used\n\t// to execute the actions, using a DBA connection, so that any\n\t// in-flight statement (e.g. an ALTER) is aborted and any\n\t// subsequent statement on the connection fails immediately.\n\t// If we don't do this then we could e.g. cause a PRS to fail as\n\t// the running ALTER will block setting [super_]read_only, or cause\n\t// a workflow delete to time out as the engine cannot process it\n\t// until the controller has stopped.\n\t// A failed/killed ALTER will be tried again when the copy","sourceCodeStart":975,"sourceCodeEnd":1011,"githubUrl":"https://github.com/vitessio/vitess/blob/01a25a7d176f94613b8d59d799f438380a8760e4/go/vt/vttablet/tabletmanager/vreplication/vreplicator.go#L975-L1011","documentation":"After executing a post copy action (typically a deferred ALTER TABLE adding secondary indexes), VReplication deletes the corresponding row from the _vt.post_copy_action table. This error wraps any MySQL failure of that DELETE, including the workflow/table name and action id for context. If it fails, the action row remains and could be re-executed on a later copy attempt.","triggerScenarios":"The deleteAction closure inside execPostCopyActions runs the bound DELETE on _vt.post_copy_action (via sqlparser.ParseAndBind + dbClient.ExecuteFetch) and the ExecuteFetch returns an error — e.g. the connection was killed, the tablet is read-only, or the table is missing/locked.","commonSituations":"Workflow interrupted mid-action (KILL of the connection due to engine shutdown or controller stop); target tablet flipped to read-only (e.g. during PRS); long-running DDL lock contention on _vt.post_copy_action; shard merges with concurrent controllers.","solutions":["Check the wrapped cause (%v suffix) for the underlying MySQL error and address it (read-only mode, connection killed, lock timeout)","Re-run/refresh the workflow — an already-executed action is detected as duplicate (ERDupKeyName handling) and skipped, or the row is cleaned up on the next copy","Verify the target tablet is writable and the _vt schema is intact (post_copy_action table exists)","If the action row is orphaned but the schema change is already applied, clean up _vt.post_copy_action manually with DBA access"],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"// Pre-check writability and row existence before retrying the workflow\nqr, err := dbaConn.ExecuteFetch(\"SELECT @@read_only, (SELECT COUNT(*) FROM _vt.post_copy_action) \", 1, false)","typeGuard":null,"tryCatchPattern":"if _, err := dbc.ExecuteFetch(delq, 1); err != nil {\n    if sqlErr, ok := err.(*sqlerror.SQLError); ok {\n        switch sqlErr.Number() {\n        case sqlerror.ERServerIsReadOnly:\n            // wait for read-only window (e.g. PRS) to end, then retry\n        case 1205: // lock wait timeout\n            // retry after competing controllers finish\n        }\n    }\n    return fmt.Errorf(\"failed to delete post copy action for the %q table with id %d: %v\", tableName, id, err)\n}","preventionTips":["Avoid stopping/deleting workflows or running PRS while copy phases with deferred indexes are in flight","Keep the target tablet writable during the copy phase","Ensure _vt.post_copy_action exists and is not locked by manual sessions","If an action already applied leaves an orphan row, clean it up via DBA access before re-running"],"tags":["vreplication","mysql","go"],"backgroundTag":"post-copy-action-delete-failed","analyzedSha":"01a25a7d176f94613b8d59d799f438380a8760e4","analyzedAt":"2026-09-01T17:28:30.605Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}