{"record":{"id":"c9d8285ef03bd5e3","repo":"vitessio/vitess","slug":"tablet-v-is-already-drained","errorCode":null,"errorMessage":"Tablet: %v, is already drained","messagePattern":"Tablet: (.+?), is already drained","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"go/vt/vttablet/tabletmanager/rpc_actions.go","lineNumber":141,"sourceCode":"func (tm *TabletManager) ChangeType(ctx context.Context, tabletType topodatapb.TabletType, semiSync bool) error {\n\tif err := tm.lock(ctx); err != nil {\n\t\treturn err\n\t}\n\tdefer tm.unlock()\n\n\tsemiSyncAction, err := tm.convertBoolToSemiSyncAction(ctx, semiSync)\n\tif err != nil {\n\t\treturn err\n\t}\n\n\treturn tm.changeTypeLocked(ctx, tabletType, DBActionNone, semiSyncAction)\n}\n\n// changeTypeLocked changes the tablet type under a lock\nfunc (tm *TabletManager) changeTypeLocked(ctx context.Context, tabletType topodatapb.TabletType, action DBAction, semiSync SemiSyncAction) error {\n\t// We don't want to allow multiple callers to claim a tablet as drained.\n\tif tabletType == topodatapb.TabletType_DRAINED && tm.Tablet().Type == topodatapb.TabletType_DRAINED {\n\t\treturn fmt.Errorf(\"Tablet: %v, is already drained\", tm.tabletAlias)\n\t}\n\n\tif err := tm.tmState.ChangeTabletType(ctx, tabletType, action); err != nil {\n\t\treturn err\n\t}\n\n\t// Let's see if we need to fix semi-sync acking.\n\tif err := tm.fixSemiSyncAndReplication(ctx, tm.Tablet().Type, semiSync); err != nil {\n\t\treturn vterrors.Wrap(err, \"fixSemiSyncAndReplication failed, may not ack correctly\")\n\t}\n\treturn nil\n}\n\n// Sleep sleeps for the duration\nfunc (tm *TabletManager) Sleep(ctx context.Context, duration time.Duration) {\n\tif err := tm.lock(ctx); err != nil {\n\t\t// client gave up\n\t\treturn","sourceCodeStart":123,"sourceCodeEnd":159,"githubUrl":"https://github.com/vitessio/vitess/blob/01a25a7d176f94613b8d59d799f438380a8760e4/go/vt/vttablet/tabletmanager/rpc_actions.go#L123-L159","documentation":"changeTypeLocked refuses to mark a tablet as DRAINED if the tablet is already in DRAINED state, to prevent multiple concurrent callers from both claiming the drained state. Callers include ChangeType, Backup, InitPrimary, InitReplica and ReplicaWasPromoted.","triggerScenarios":"Two concurrent operations that both need to drain the tablet (e.g. two backups, or a backup racing a type change), or a stale tablet record still showing DRAINED from a previously crashed operation.","commonSituations":"Concurrent vtctldclient commands against the same tablet, a backup running while another maintenance operation starts, leftover DRAINED state after a vttablet crash mid-operation.","solutions":["Wait for the currently running draining operation to finish and release the tablet","Restart vttablet to clear a stale DRAINED type left by a crashed operation","Check the tablet's current type (vtctldclient GetTablet) before issuing another drain-requiring operation","Serialize maintenance operations on a tablet instead of running them concurrently"],"exampleFix":null,"handlingStrategy":"retry","validationCode":"// Check the tablet is not already drained before requesting a drain-requiring operation\ntab := getTablet(t, alias)\nif tab.Type == \"DRAINED\" {\n    return fmt.Errorf(\"tablet %s already drained; wait or restart vttablet\", alias)\n}","typeGuard":null,"tryCatchPattern":"err := withRetry(ctx, func() error { return vtctldclientBackup(alias) })\nif err != nil && strings.Contains(err.Error(), \"is already drained\") {\n    // wait for the current operation, then retry\n}","preventionTips":["Serialize maintenance operations per tablet (no concurrent backups/type changes)","Check tablet type with vtctldclient GetTablet before maintenance","Restart vttablet if DRAINED state is stale after a crash","Automate locks/leases for drain-requiring operations"],"tags":["tablet","drained","concurrency","tablet-type"],"backgroundTag":"tablet-already-drained","analyzedSha":"01a25a7d176f94613b8d59d799f438380a8760e4","analyzedAt":"2026-09-01T17:28:30.605Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}