{"record":{"id":"89e0f4c0a5559080","repo":"vitessio/vitess","slug":"a-backup-is-already-running-on-tablet-v","errorCode":null,"errorMessage":"a backup is already running on tablet: %v","messagePattern":"a backup is already running on tablet: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"go/vt/vttablet/tabletmanager/rpc_backup.go","lineNumber":241,"sourceCode":"\t// Create the logger: tee to console and source.\n\tl := logutil.NewTeeLogger(logutil.NewConsoleLogger(), logger)\n\n\t// Now we can run restore.\n\tstartTime = time.Now()\n\tbackupEngine, restoreErr = tm.restoreBackupLocked(ctx, l, 0 /* waitForBackupInterval */, true /* deleteBeforeRestore */, request, mysqlShutdownTimeout)\n\n\treturn restoreErr\n}\n\nfunc (tm *TabletManager) IsBackupRunning() bool {\n\treturn tm._isBackupRunning\n}\n\nfunc (tm *TabletManager) beginBackup(backupMode string) error {\n\ttm.mutex.Lock()\n\tdefer tm.mutex.Unlock()\n\tif tm._isBackupRunning {\n\t\treturn fmt.Errorf(\"a backup is already running on tablet: %v\", tm.tabletAlias)\n\t}\n\t// When mode is online we don't take the action lock, so we continue to serve,\n\t// but let's set _isBackupRunning to true.\n\t// So that we only allow one online backup at a time.\n\t// Offline backups also run only one at a time because we take the action lock\n\t// so this is not really needed in that case, however we are using it to record the state\n\ttm._isBackupRunning = true\n\tstatsBackupIsRunning.Set([]string{backupMode}, 1)\n\treturn nil\n}\n\nfunc (tm *TabletManager) endBackup(backupMode string) {\n\t// Now we set _isBackupRunning back to false.\n\t// Have to take the mutex lock before writing to _ fields.\n\ttm.mutex.Lock()\n\tdefer tm.mutex.Unlock()\n\ttm._isBackupRunning = false\n\tstatsBackupIsRunning.Set([]string{backupMode}, 0)","sourceCodeStart":223,"sourceCodeEnd":259,"githubUrl":"https://github.com/vitessio/vitess/blob/01a25a7d176f94613b8d59d799f438380a8760e4/go/vt/vttablet/tabletmanager/rpc_backup.go#L223-L259","documentation":"beginBackup enforces that only one backup runs at a time on a tablet, using the _isBackupRunning flag guarded by the tablet manager mutex. If a backup (online or offline) is already in progress, any new Backup RPC fails immediately with this error.","triggerScenarios":"Calling vtctldclient Backup (or an automated backup schedule) on a tablet while a previous backup — including an online backup that does not take the action lock — is still running.","commonSituations":"Overlapping scheduled backups, a hung/slow backup still running when a manual backup is triggered, retry loops re-triggering Backup before the first attempt finished.","solutions":["Wait for the running backup to complete (check vttablet logs for backup progress)","Cancel or kill the stuck backup and its process, then retry","Stagger scheduled backups so they do not overlap on the same tablet","Restart vttablet if the backup process died and left the flag set (the in-memory flag is cleared on restart)"],"exampleFix":null,"handlingStrategy":"retry","validationCode":"// Guard: only start a backup when none is in flight on this tablet\nif backupInFlight[alias] {\n    return fmt.Errorf(\"backup already scheduled for %s\", alias)\n}","typeGuard":null,"tryCatchPattern":"err := vtctldclientBackup(alias)\nif err != nil && strings.Contains(err.Error(), \"a backup is already running\") {\n    time.Sleep(backoff)\n    return vtctldclientBackup(alias) // retry after the running backup finishes\n}","preventionTips":["Schedule backups so they never overlap on the same tablet","Check vttablet logs for an active backup before manual triggers","Use exponential backoff on retries","Alert on backups exceeding expected duration to catch hung ones"],"tags":["backup","tablet","concurrency"],"backgroundTag":"backup-already-running","analyzedSha":"01a25a7d176f94613b8d59d799f438380a8760e4","analyzedAt":"2026-09-01T17:28:30.605Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}