What causes "shard has abandon backup operation" in weaviate/weaviate?

Retry the whole backup operation; the stale commit cannot be accepted

shard has abandon backup operation

Error message

shard has abandon backup operation

What it means

OnCommit is how the coordinator confirms a pending shard backup operation. If the shard's last recorded operation ID does not match the commit's ID, or the shard is no longer in the waiting-for-coordinator state, the commit is rejected — the shard considers the backup operation abandoned (timed out, aborted, or already finished).

Source

Thrown at usecases/backup/shard.go:234

						}).Warn("received abort request for different backup ID, ignoring")
					}
				}
			case <-done: // caller is done
				return
			}
		}
	}, logger)
	return ctx
}

// OnCommit will be triggered when the coordinator confirms the execution of a previous operation
func (c *shardSyncChan) OnCommit(ctx context.Context, req *StatusRequest) error {
	st := c.lastOp.get()
	if st.ID == req.ID && c.waitingForCoordinatorToCommit.Load() {
		c.coordChan <- *req
		return nil
	}
	return fmt.Errorf("shard has abandon backup operation")
}

// Abort tells a node to abort the previous backup operation
func (c *shardSyncChan) OnAbort(_ context.Context, req *AbortRequest) error {
	st := c.lastOp.get()
	if st.ID == req.ID {
		c.coordChan <- *req
		return nil
	}
	// No active operation with this ID - this is not an error, the operation may have
	// already completed or never started on this node. Return nil for idempotency.
	return nil
}

View on GitHub (pinned to 75aa4b6d11)

Solutions

Retry the whole backup operation; the stale commit cannot be accepted
Check for coordinator slowness or restarts that delay commits past the shard timeout
Increase coordination timeouts if commits routinely arrive late
Verify only one backup operation per shard is running at a time

Defensive patterns

Strategy: retry

Try / catch

if err := confirmOrCommit(ctx, req); err != nil {
    if strings.Contains(err.Error(), "shard has abandon backup operation") {
        // stale commit: the shard already gave up; restart the whole operation
        restartBackupOperation(ctx, req)
    }
}

Prevention

Keep coordinator commit latency well under shard wait timeouts
Avoid concurrent backup/restore operations on the same shard
Prevent node restarts during an active backup
Alert on slow coordinator loops (store latency, GC pauses)

When it happens

Trigger: Coordinator sends StatusRequest commit for a backup ID the shard no longer tracks: the shard's wait timed out, the op already completed, or a stale/duplicate commit arrived after the shard cleared its waitingForCoordinatorToCommit flag.

Common situations: Commit arriving after waitForCoordinator timed out (slow coordinator); retried commit messages racing with a completed op; coordinator and shard clocks/timeouts badly mismatched; shard restarted between the wait and the commit.

Related errors

AI-assisted analysis of weaviate/weaviate@75aa4b6d11 (2026-09-04). Data as JSON: /api/errors/3024a437885b9a2a. Report an issue: GitHub.