What causes "obtainQueueLock: empty queue node: %v" in vitessio/vitess?

Retry the operation — the lock loop usually recovers once a fresh lock node is created

obtainQueueLock: empty queue node: %v

Error message

obtainQueueLock: empty queue node: %v

What it means

obtainQueueLock implements zk queue-based locking: it lists children of the queue node and the lexically smallest child holds the lock. If the queue node unexpectedly has zero children — it may have just been created without a lock node, or children vanished between creation and listing — it errors rather than proceeding, since the lock state is inconsistent.

Source

Thrown at go/vt/topo/zk2topo/utils.go:305

// obtainQueueLock waits until we hold the lock in the provided path.
// The lexically lowest node is the lock holder - verify that this
// path holds the lock.  Call this queue-lock because the semantics are
// a hybrid.  Normal Zookeeper locks make assumptions about sequential
// numbering that don't hold when the data in a lock is modified.
func obtainQueueLock(ctx context.Context, conn *ZkConn, zkPath string) error {
	queueNode := path.Dir(zkPath)
	lockNode := path.Base(zkPath)

	for {
		// Get our siblings.
		children, _, err := conn.Children(ctx, queueNode)
		if err != nil {
			return vterrors.Wrap(err, "obtainQueueLock: trylock failed %v")
		}
		sort.Strings(children)
		if len(children) == 0 {
			return fmt.Errorf("obtainQueueLock: empty queue node: %v", queueNode)
		}

		// If we are the first node, we got the lock.
		if children[0] == lockNode {
			return nil
		}

		// If not, find the previous node.
		prevLock := ""
		for i := 1; i < len(children); i++ {
			if children[i] == lockNode {
				prevLock = children[i-1]
				break
			}
		}
		if prevLock == "" {
			return fmt.Errorf("obtainQueueLock: no previous queue node found: %v", zkPath)
		}

View on GitHub (pinned to 01a25a7d17)

Solutions

Retry the operation — the lock loop usually recovers once a fresh lock node is created
Check for zookeeper session expiry in logs (session timeout too short for the operation)
Delete and recreate the empty queue node: run the zkctl/zk utility to rm the path so lock can recreate it with a lock child
Ensure only one leadership process runs per queue path to reduce race churn

Defensive patterns

Strategy: retry

Validate before calling

children, _, err := conn.Children(ctx, queueNode)
if err == nil && len(children) == 0 { /* queue node is empty; recreate before locking */ }

Type guard

func isQueueLockRace(err error) bool {
    return err != nil && strings.Contains(err.Error(), "empty queue node")
}

Try / catch

err := zk2topo.WaitForLeadership(ctx, ts, path, action)
if isQueueLockRace(err) {
    // retry; the lock loop recreates its ephemeral node
}

Prevention

Keep zookeeper session timeouts long enough to avoid ephemeral node loss mid-lock
Avoid multiple leadership processes on the same queue path
Clean up stale/empty queue directories after crashed processes
Retry leadership acquisition with backoff

When it happens

Trigger: WaitForLeadership or lock on a zk queue path where the node exists but has no children — e.g. the lock node's ephemeral creation failed or expired before Children was called, or two waiters raced and the winner's node disappeared between the list and comparison, leading to a retry loop that lands on an empty node.

Common situations: Zookeeper session expiry dropping ephemeral lock nodes mid-election; two processes competing for leadership of the same topo path; stale/empty queue directories left over from crashed processes.

Related errors

AI-assisted analysis of vitessio/vitess@01a25a7d17 (2026-09-01). Data as JSON: /api/errors/f094550b0809c852. Report an issue: GitHub.