DeleteRecursive: nodes getting recreated underneath delete…

Error message

DeleteRecursive: nodes getting recreated underneath delete (app race condition): %v

What it means

DeleteRecursive finished deleting a zk node's children and tried to delete the node itself, but the final Delete failed. Because the failure is not ErrNotEmpty, zk2topo reinterprets it as an 'app race condition': something is recreating nodes underneath the path being deleted. The original underlying error (e.g. NoNode, connection loss) is replaced by this message, which can mask the true cause.

Solutions

Re-run DeleteRecursive after the competing writer stops (often the second run succeeds or returns NoNode which is success-equivalent)
Identify and stop the process recreating nodes under the path (check ephemeral owners / watches)
Retry with backoff, treating this error as transient
If it is actually a NoNode, verify the path is gone — the delete already effectively completed

Example fix

// before
zk2topo.DeleteRecursive(ctx, ts, path) // single shot
// after
for i := 0; i < 3; i++ {
    err := zk2topo.DeleteRecursive(ctx, ts, path)
    if err == nil || topo.IsErrType(err, topo.NoNode) {
        break
    }
    time.Sleep(2 * time.Second)
}

Defensive patterns

Strategy: retry

Validate before calling

// check no active ephemeral children before deleting
children, _, err := conn.Children(ctx, path)
if err == nil && len(children) > 0 { /* writers present; defer delete */ }

Type guard

func isDeleteRace(err error) bool {
    return err != nil && strings.Contains(err.Error(), "nodes getting recreated underneath delete")
}

Try / catch

err := zk2topo.DeleteRecursive(ctx, ts, path)
if isDeleteRace(err) {
    // retry with backoff; a later run usually returns NoNode (success)
}

Prevention

Stop competing writers/services before recursive deletes
Use distributed locks around shared topo cleanup
Treat this error as transient and retry with backoff
Check for ephemeral node owners if deletions keep failing

When it happens

Trigger: Calling DeleteRecursive (directly or via commandRm) on a zk path while another process recreates children concurrently, or when the final Delete fails with an unexpected error such as the node already being deleted (NoNode) by a concurrent deleter.

Common situations: Two operators/automation running vtctldclient Rm on the same path; a service re-registering ephemeral nodes during cleanup; zookeeper session churn causing NoNode on the final delete; recursive retry loops hitting the same race.

Related errors

AI-assisted analysis of vitessio/vitess@01a25a7d17 (2026-09-01). Data as JSON: /api/errors/99335965b6938e96. Report an issue: GitHub.

Appendix: source

Thrown at go/vt/topo/zk2topo/utils.go:283

	// Otherwise, you can enter a race condition, or get starved out from deleting.
	err = zconn.SetACL(ctx, zkPath, zk.WorldACL(zk.PermAdmin|zk.PermDelete|zk.PermRead), version)
	if err != nil {
		return err
	}
	children, _, err := zconn.Children(ctx, zkPath)
	if err != nil {
		return err
	}
	for _, child := range children {
		err := DeleteRecursive(ctx, zconn, path.Join(zkPath, child), -1)
		if err != nil && err != zk.ErrNoNode {
			return vterrors.Wrapf(err, "DeleteRecursive: recursive delete failed")
		}
	}

	err = zconn.Delete(ctx, zkPath, version)
	if err != nil && err != zk.ErrNotEmpty {
		err = fmt.Errorf("DeleteRecursive: nodes getting recreated underneath delete (app race condition): %v", zkPath)
	}
	return err
}

// obtainQueueLock waits until we hold the lock in the provided path.
// The lexically lowest node is the lock holder - verify that this
// path holds the lock.  Call this queue-lock because the semantics are
// a hybrid.  Normal Zookeeper locks make assumptions about sequential
// numbering that don't hold when the data in a lock is modified.
func obtainQueueLock(ctx context.Context, conn *ZkConn, zkPath string) error {
	queueNode := path.Dir(zkPath)
	lockNode := path.Base(zkPath)

	for {
		// Get our siblings.
		children, _, err := conn.Children(ctx, queueNode)
		if err != nil {
			return vterrors.Wrap(err, "obtainQueueLock: trylock failed %v")

View on GitHub (pinned to 01a25a7d17)