{"record":{"id":"3024a437885b9a2a","repo":"weaviate/weaviate","slug":"shard-has-abandon-backup-operation","errorCode":null,"errorMessage":"shard has abandon backup operation","messagePattern":"shard has abandon backup operation","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"usecases/backup/shard.go","lineNumber":234,"sourceCode":"\t\t\t\t\t\t}).Warn(\"received abort request for different backup ID, ignoring\")\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\tcase <-done: // caller is done\n\t\t\t\treturn\n\t\t\t}\n\t\t}\n\t}, logger)\n\treturn ctx\n}\n\n// OnCommit will be triggered when the coordinator confirms the execution of a previous operation\nfunc (c *shardSyncChan) OnCommit(ctx context.Context, req *StatusRequest) error {\n\tst := c.lastOp.get()\n\tif st.ID == req.ID && c.waitingForCoordinatorToCommit.Load() {\n\t\tc.coordChan <- *req\n\t\treturn nil\n\t}\n\treturn fmt.Errorf(\"shard has abandon backup operation\")\n}\n\n// Abort tells a node to abort the previous backup operation\nfunc (c *shardSyncChan) OnAbort(_ context.Context, req *AbortRequest) error {\n\tst := c.lastOp.get()\n\tif st.ID == req.ID {\n\t\tc.coordChan <- *req\n\t\treturn nil\n\t}\n\t// No active operation with this ID - this is not an error, the operation may have\n\t// already completed or never started on this node. Return nil for idempotency.\n\treturn nil\n}\n","sourceCodeStart":216,"sourceCodeEnd":248,"githubUrl":"https://github.com/weaviate/weaviate/blob/75aa4b6d11f8818305aafd4440b4e32794f7ca04/usecases/backup/shard.go#L216-L248","documentation":"OnCommit is how the coordinator confirms a pending shard backup operation. If the shard's last recorded operation ID does not match the commit's ID, or the shard is no longer in the waiting-for-coordinator state, the commit is rejected — the shard considers the backup operation abandoned (timed out, aborted, or already finished).","triggerScenarios":"Coordinator sends StatusRequest commit for a backup ID the shard no longer tracks: the shard's wait timed out, the op already completed, or a stale/duplicate commit arrived after the shard cleared its waitingForCoordinatorToCommit flag.","commonSituations":"Commit arriving after waitForCoordinator timed out (slow coordinator); retried commit messages racing with a completed op; coordinator and shard clocks/timeouts badly mismatched; shard restarted between the wait and the commit.","solutions":["Retry the whole backup operation; the stale commit cannot be accepted","Check for coordinator slowness or restarts that delay commits past the shard timeout","Increase coordination timeouts if commits routinely arrive late","Verify only one backup operation per shard is running at a time"],"exampleFix":null,"handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"if err := confirmOrCommit(ctx, req); err != nil {\n    if strings.Contains(err.Error(), \"shard has abandon backup operation\") {\n        // stale commit: the shard already gave up; restart the whole operation\n        restartBackupOperation(ctx, req)\n    }\n}","preventionTips":["Keep coordinator commit latency well under shard wait timeouts","Avoid concurrent backup/restore operations on the same shard","Prevent node restarts during an active backup","Alert on slow coordinator loops (store latency, GC pauses)"],"tags":["backup","distributed","coordinator","state-mismatch"],"backgroundTag":"backup-operation-abandoned","analyzedSha":"75aa4b6d11f8818305aafd4440b4e32794f7ca04","analyzedAt":"2026-09-04T14:58:20.392Z","contentChangedAt":"2026-09-04T14:58:20.392Z","schemaVersion":2},"datasetVersion":"2026-09-11T21:17:09.523Z"}