vitessio/vitess · error

obtainQueueLock: no previous queue node found: %v

Error message

obtainQueueLock: no previous queue node found: %v

What it means

obtainQueueLock implements a ZooKeeper lock queue: after creating this client's sequential lock node, it scans the sorted children of the queue directory to find the node created immediately before its own. If the client's lockNode is not found among the children (or is the first entry so no predecessor exists), the invariant of the queue is broken and it returns this error instead of setting a watch on a non-existent predecessor.

Source

Thrown at go/vt/topo/zk2topo/utils.go:322

		if len(children) == 0 {
			return fmt.Errorf("obtainQueueLock: empty queue node: %v", queueNode)
		}

		// If we are the first node, we got the lock.
		if children[0] == lockNode {
			return nil
		}

		// If not, find the previous node.
		prevLock := ""
		for i := 1; i < len(children); i++ {
			if children[i] == lockNode {
				prevLock = children[i-1]
				break
			}
		}
		if prevLock == "" {
			return fmt.Errorf("obtainQueueLock: no previous queue node found: %v", zkPath)
		}

		// Set a watch on the previous node.
		zkPrevLock := path.Join(queueNode, prevLock)
		exists, _, watch, err := conn.ExistsW(ctx, zkPrevLock)
		if err != nil {
			return vterrors.Wrapf(err, "obtainQueueLock: unable to watch queued node %v", zkPrevLock)
		}
		if !exists {
			// The lock disappeared, try to read again.
			continue
		}
		select {
		case <-ctx.Done():
			return ctx.Err()
		case <-watch:
			// Something happened to the previous lock.
			// It doesn't matter what, read again.

View on GitHub (pinned to 01a25a7d17)

Solutions

  1. Retry the leadership/lock acquisition: the ephemeral node is gone, so a fresh obtainQueueLock call will recreate the sequential node and succeed.
  2. Check ZooKeeper session health (logs for session expired/Disconnected events) and network stability between the client and the ZK ensemble.
  3. Verify no external scripts or cleanup jobs are deleting nodes under the queue directory while elections are in progress.
  4. Increase ZooKeeper session timeout / tune session tick settings if ephemeral nodes expire frequently under load.

Example fix

// before: single attempt, hard failure
if err := ts.WaitForLeadership(ctx, ...); err != nil {
    return err
}
// after: retry on transient queue inconsistencies
for i := 0; i < 3; i++ {
    err := ts.WaitForLeadership(ctx, ...)
    if err == nil {
        return nil
    }
    if strings.Contains(err.Error(), "no previous queue node found") {
        time.Sleep(time.Second)
        continue
    }
    return err
}
return err
Defensive patterns

Strategy: retry

Validate before calling

kids, err := conn.Children(ctx, zkPath)
if err == nil && len(kids) == 0 {
    // queue empty; safe to (re)create lock node, no predecessor expected
}

Type guard

func isNoPrevQueueNode(err error) bool { return err != nil && strings.Contains(err.Error(), "no previous queue node found") }

Try / catch

if err := ts.WaitForLeadership(ctx, shardInfo); err != nil {
    if isNoPrevQueueNode(err) {
        // transient: session lost ephemeral node; retry with backoff
        return retryWithBackoff(ctx, ts.WaitForLeadership, shardInfo)
    }
    return err
}

Prevention

When it happens

Trigger: Calling topo.Server.WaitForLeadership or the internal lock path on a zk2 topo server when the sequential ephemeral node created by Create() is missing from the queue directory listing — e.g. the ephemeral node expired (session loss) between creation and the Children() listing, or a concurrent cleanup deleted queue nodes.

Common situations: ZooKeeper session timeouts/expirations under network partitions or GC pauses that kill the ephemeral lock node; running vtctld/vttablet against a ZooKeeper ensemble being restarted; leftovers or manual deletion of entries under /vitess/<cell>/action or the election queue path.

Related errors

AI-assisted analysis of vitessio/vitess@01a25a7d17 (2026-09-01). Data as JSON: /api/errors/fc48ca2322530333. Report an issue: GitHub.