{"record":{"id":"99335965b6938e96","repo":"vitessio/vitess","slug":"deleterecursive-nodes-getting-recreated-underneat","errorCode":null,"errorMessage":"DeleteRecursive: nodes getting recreated underneath delete (app race condition): %v","messagePattern":"DeleteRecursive: nodes getting recreated underneath delete \\(app race condition\\): (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"go/vt/topo/zk2topo/utils.go","lineNumber":283,"sourceCode":"\t// Otherwise, you can enter a race condition, or get starved out from deleting.\n\terr = zconn.SetACL(ctx, zkPath, zk.WorldACL(zk.PermAdmin|zk.PermDelete|zk.PermRead), version)\n\tif err != nil {\n\t\treturn err\n\t}\n\tchildren, _, err := zconn.Children(ctx, zkPath)\n\tif err != nil {\n\t\treturn err\n\t}\n\tfor _, child := range children {\n\t\terr := DeleteRecursive(ctx, zconn, path.Join(zkPath, child), -1)\n\t\tif err != nil && err != zk.ErrNoNode {\n\t\t\treturn vterrors.Wrapf(err, \"DeleteRecursive: recursive delete failed\")\n\t\t}\n\t}\n\n\terr = zconn.Delete(ctx, zkPath, version)\n\tif err != nil && err != zk.ErrNotEmpty {\n\t\terr = fmt.Errorf(\"DeleteRecursive: nodes getting recreated underneath delete (app race condition): %v\", zkPath)\n\t}\n\treturn err\n}\n\n// obtainQueueLock waits until we hold the lock in the provided path.\n// The lexically lowest node is the lock holder - verify that this\n// path holds the lock.  Call this queue-lock because the semantics are\n// a hybrid.  Normal Zookeeper locks make assumptions about sequential\n// numbering that don't hold when the data in a lock is modified.\nfunc obtainQueueLock(ctx context.Context, conn *ZkConn, zkPath string) error {\n\tqueueNode := path.Dir(zkPath)\n\tlockNode := path.Base(zkPath)\n\n\tfor {\n\t\t// Get our siblings.\n\t\tchildren, _, err := conn.Children(ctx, queueNode)\n\t\tif err != nil {\n\t\t\treturn vterrors.Wrap(err, \"obtainQueueLock: trylock failed %v\")","sourceCodeStart":265,"sourceCodeEnd":301,"githubUrl":"https://github.com/vitessio/vitess/blob/01a25a7d176f94613b8d59d799f438380a8760e4/go/vt/topo/zk2topo/utils.go#L265-L301","documentation":"DeleteRecursive finished deleting a zk node's children and tried to delete the node itself, but the final Delete failed. Because the failure is not ErrNotEmpty, zk2topo reinterprets it as an 'app race condition': something is recreating nodes underneath the path being deleted. The original underlying error (e.g. NoNode, connection loss) is replaced by this message, which can mask the true cause.","triggerScenarios":"Calling DeleteRecursive (directly or via commandRm) on a zk path while another process recreates children concurrently, or when the final Delete fails with an unexpected error such as the node already being deleted (NoNode) by a concurrent deleter.","commonSituations":"Two operators/automation running vtctldclient Rm on the same path; a service re-registering ephemeral nodes during cleanup; zookeeper session churn causing NoNode on the final delete; recursive retry loops hitting the same race.","solutions":["Re-run DeleteRecursive after the competing writer stops (often the second run succeeds or returns NoNode which is success-equivalent)","Identify and stop the process recreating nodes under the path (check ephemeral owners / watches)","Retry with backoff, treating this error as transient","If it is actually a NoNode, verify the path is gone — the delete already effectively completed"],"exampleFix":"// before\nzk2topo.DeleteRecursive(ctx, ts, path) // single shot\n// after\nfor i := 0; i < 3; i++ {\n    err := zk2topo.DeleteRecursive(ctx, ts, path)\n    if err == nil || topo.IsErrType(err, topo.NoNode) {\n        break\n    }\n    time.Sleep(2 * time.Second)\n}","handlingStrategy":"retry","validationCode":"// check no active ephemeral children before deleting\nchildren, _, err := conn.Children(ctx, path)\nif err == nil && len(children) > 0 { /* writers present; defer delete */ }","typeGuard":"func isDeleteRace(err error) bool {\n    return err != nil && strings.Contains(err.Error(), \"nodes getting recreated underneath delete\")\n}","tryCatchPattern":"err := zk2topo.DeleteRecursive(ctx, ts, path)\nif isDeleteRace(err) {\n    // retry with backoff; a later run usually returns NoNode (success)\n}","preventionTips":["Stop competing writers/services before recursive deletes","Use distributed locks around shared topo cleanup","Treat this error as transient and retry with backoff","Check for ephemeral node owners if deletions keep failing"],"tags":["zookeeper","topo","concurrency","race-condition"],"backgroundTag":"zookeeper-node-race","analyzedSha":"01a25a7d176f94613b8d59d799f438380a8760e4","analyzedAt":"2026-09-01T17:28:30.605Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}