{"record":{"id":"fc48ca2322530333","repo":"vitessio/vitess","slug":"obtainqueuelock-no-previous-queue-node-found-v","errorCode":null,"errorMessage":"obtainQueueLock: no previous queue node found: %v","messagePattern":"obtainQueueLock: no previous queue node found: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"go/vt/topo/zk2topo/utils.go","lineNumber":322,"sourceCode":"\t\tif len(children) == 0 {\n\t\t\treturn fmt.Errorf(\"obtainQueueLock: empty queue node: %v\", queueNode)\n\t\t}\n\n\t\t// If we are the first node, we got the lock.\n\t\tif children[0] == lockNode {\n\t\t\treturn nil\n\t\t}\n\n\t\t// If not, find the previous node.\n\t\tprevLock := \"\"\n\t\tfor i := 1; i < len(children); i++ {\n\t\t\tif children[i] == lockNode {\n\t\t\t\tprevLock = children[i-1]\n\t\t\t\tbreak\n\t\t\t}\n\t\t}\n\t\tif prevLock == \"\" {\n\t\t\treturn fmt.Errorf(\"obtainQueueLock: no previous queue node found: %v\", zkPath)\n\t\t}\n\n\t\t// Set a watch on the previous node.\n\t\tzkPrevLock := path.Join(queueNode, prevLock)\n\t\texists, _, watch, err := conn.ExistsW(ctx, zkPrevLock)\n\t\tif err != nil {\n\t\t\treturn vterrors.Wrapf(err, \"obtainQueueLock: unable to watch queued node %v\", zkPrevLock)\n\t\t}\n\t\tif !exists {\n\t\t\t// The lock disappeared, try to read again.\n\t\t\tcontinue\n\t\t}\n\t\tselect {\n\t\tcase <-ctx.Done():\n\t\t\treturn ctx.Err()\n\t\tcase <-watch:\n\t\t\t// Something happened to the previous lock.\n\t\t\t// It doesn't matter what, read again.","sourceCodeStart":304,"sourceCodeEnd":340,"githubUrl":"https://github.com/vitessio/vitess/blob/01a25a7d176f94613b8d59d799f438380a8760e4/go/vt/topo/zk2topo/utils.go#L304-L340","documentation":"obtainQueueLock implements a ZooKeeper lock queue: after creating this client's sequential lock node, it scans the sorted children of the queue directory to find the node created immediately before its own. If the client's lockNode is not found among the children (or is the first entry so no predecessor exists), the invariant of the queue is broken and it returns this error instead of setting a watch on a non-existent predecessor.","triggerScenarios":"Calling topo.Server.WaitForLeadership or the internal lock path on a zk2 topo server when the sequential ephemeral node created by Create() is missing from the queue directory listing — e.g. the ephemeral node expired (session loss) between creation and the Children() listing, or a concurrent cleanup deleted queue nodes.","commonSituations":"ZooKeeper session timeouts/expirations under network partitions or GC pauses that kill the ephemeral lock node; running vtctld/vttablet against a ZooKeeper ensemble being restarted; leftovers or manual deletion of entries under /vitess/<cell>/action or the election queue path.","solutions":["Retry the leadership/lock acquisition: the ephemeral node is gone, so a fresh obtainQueueLock call will recreate the sequential node and succeed.","Check ZooKeeper session health (logs for session expired/Disconnected events) and network stability between the client and the ZK ensemble.","Verify no external scripts or cleanup jobs are deleting nodes under the queue directory while elections are in progress.","Increase ZooKeeper session timeout / tune session tick settings if ephemeral nodes expire frequently under load."],"exampleFix":"// before: single attempt, hard failure\nif err := ts.WaitForLeadership(ctx, ...); err != nil {\n    return err\n}\n// after: retry on transient queue inconsistencies\nfor i := 0; i < 3; i++ {\n    err := ts.WaitForLeadership(ctx, ...)\n    if err == nil {\n        return nil\n    }\n    if strings.Contains(err.Error(), \"no previous queue node found\") {\n        time.Sleep(time.Second)\n        continue\n    }\n    return err\n}\nreturn err","handlingStrategy":"retry","validationCode":"kids, err := conn.Children(ctx, zkPath)\nif err == nil && len(kids) == 0 {\n    // queue empty; safe to (re)create lock node, no predecessor expected\n}","typeGuard":"func isNoPrevQueueNode(err error) bool { return err != nil && strings.Contains(err.Error(), \"no previous queue node found\") }","tryCatchPattern":"if err := ts.WaitForLeadership(ctx, shardInfo); err != nil {\n    if isNoPrevQueueNode(err) {\n        // transient: session lost ephemeral node; retry with backoff\n        return retryWithBackoff(ctx, ts.WaitForLeadership, shardInfo)\n    }\n    return err\n}","preventionTips":["Keep ZooKeeper sessions healthy: tune session timeouts and avoid long GC pauses that trigger session expiry","Never manually delete nodes under the election/action queue directories while instances are electing","Wrap leadership acquisition in retry-with-backoff since ephemeral node loss is transient"],"tags":["zookeeper","topo","leadership-election","locking"],"backgroundTag":"zk-ephemeral-node-lost","analyzedSha":"01a25a7d176f94613b8d59d799f438380a8760e4","analyzedAt":"2026-09-01T17:28:30.605Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}