{"record":{"id":"f094550b0809c852","repo":"vitessio/vitess","slug":"obtainqueuelock-empty-queue-node-v","errorCode":null,"errorMessage":"obtainQueueLock: empty queue node: %v","messagePattern":"obtainQueueLock: empty queue node: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"go/vt/topo/zk2topo/utils.go","lineNumber":305,"sourceCode":"\n// obtainQueueLock waits until we hold the lock in the provided path.\n// The lexically lowest node is the lock holder - verify that this\n// path holds the lock.  Call this queue-lock because the semantics are\n// a hybrid.  Normal Zookeeper locks make assumptions about sequential\n// numbering that don't hold when the data in a lock is modified.\nfunc obtainQueueLock(ctx context.Context, conn *ZkConn, zkPath string) error {\n\tqueueNode := path.Dir(zkPath)\n\tlockNode := path.Base(zkPath)\n\n\tfor {\n\t\t// Get our siblings.\n\t\tchildren, _, err := conn.Children(ctx, queueNode)\n\t\tif err != nil {\n\t\t\treturn vterrors.Wrap(err, \"obtainQueueLock: trylock failed %v\")\n\t\t}\n\t\tsort.Strings(children)\n\t\tif len(children) == 0 {\n\t\t\treturn fmt.Errorf(\"obtainQueueLock: empty queue node: %v\", queueNode)\n\t\t}\n\n\t\t// If we are the first node, we got the lock.\n\t\tif children[0] == lockNode {\n\t\t\treturn nil\n\t\t}\n\n\t\t// If not, find the previous node.\n\t\tprevLock := \"\"\n\t\tfor i := 1; i < len(children); i++ {\n\t\t\tif children[i] == lockNode {\n\t\t\t\tprevLock = children[i-1]\n\t\t\t\tbreak\n\t\t\t}\n\t\t}\n\t\tif prevLock == \"\" {\n\t\t\treturn fmt.Errorf(\"obtainQueueLock: no previous queue node found: %v\", zkPath)\n\t\t}","sourceCodeStart":287,"sourceCodeEnd":323,"githubUrl":"https://github.com/vitessio/vitess/blob/01a25a7d176f94613b8d59d799f438380a8760e4/go/vt/topo/zk2topo/utils.go#L287-L323","documentation":"obtainQueueLock implements zk queue-based locking: it lists children of the queue node and the lexically smallest child holds the lock. If the queue node unexpectedly has zero children — it may have just been created without a lock node, or children vanished between creation and listing — it errors rather than proceeding, since the lock state is inconsistent.","triggerScenarios":"WaitForLeadership or lock on a zk queue path where the node exists but has no children — e.g. the lock node's ephemeral creation failed or expired before Children was called, or two waiters raced and the winner's node disappeared between the list and comparison, leading to a retry loop that lands on an empty node.","commonSituations":"Zookeeper session expiry dropping ephemeral lock nodes mid-election; two processes competing for leadership of the same topo path; stale/empty queue directories left over from crashed processes.","solutions":["Retry the operation — the lock loop usually recovers once a fresh lock node is created","Check for zookeeper session expiry in logs (session timeout too short for the operation)","Delete and recreate the empty queue node: run the zkctl/zk utility to rm the path so lock can recreate it with a lock child","Ensure only one leadership process runs per queue path to reduce race churn"],"exampleFix":null,"handlingStrategy":"retry","validationCode":"children, _, err := conn.Children(ctx, queueNode)\nif err == nil && len(children) == 0 { /* queue node is empty; recreate before locking */ }","typeGuard":"func isQueueLockRace(err error) bool {\n    return err != nil && strings.Contains(err.Error(), \"empty queue node\")\n}","tryCatchPattern":"err := zk2topo.WaitForLeadership(ctx, ts, path, action)\nif isQueueLockRace(err) {\n    // retry; the lock loop recreates its ephemeral node\n}","preventionTips":["Keep zookeeper session timeouts long enough to avoid ephemeral node loss mid-lock","Avoid multiple leadership processes on the same queue path","Clean up stale/empty queue directories after crashed processes","Retry leadership acquisition with backoff"],"tags":["zookeeper","topo","locking","race-condition"],"backgroundTag":"zookeeper-node-race","analyzedSha":"01a25a7d176f94613b8d59d799f438380a8760e4","analyzedAt":"2026-09-01T17:28:30.605Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}