{"record":{"id":"792037ceb6174f5d","repo":"hashicorp/nomad","slug":"variable-already-holds-a-lock","errorCode":null,"errorMessage":"variable already holds a lock","messagePattern":"variable already holds a lock","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"nomad/state/state_store_variables.go","lineNumber":16,"sourceCode":"// Copyright IBM Corp. 2015, 2026\n// SPDX-License-Identifier: BUSL-1.1\n\npackage state\n\nimport (\n\t\"errors\"\n\t\"fmt\"\n\t\"math\"\n\n\t\"github.com/hashicorp/go-memdb\"\n\t\"github.com/hashicorp/nomad/nomad/structs\"\n)\n\nvar (\n\terrVarAlreadyLocked = errors.New(\"variable already holds a lock\")\n\terrVarNotFound      = errors.New(\"variable doesn't exist\")\n\terrLockNotFound     = errors.New(\"variable doesn't hold a lock\")\n)\n\n// Variables queries all the variables and is used only for\n// snapshot/restore and key rotation\nfunc (s *StateStore) Variables(ws memdb.WatchSet) (memdb.ResultIterator, error) {\n\ttxn := s.db.ReadTxn()\n\n\titer, err := txn.Get(TableVariables, indexID)\n\tif err != nil {\n\t\treturn nil, err\n\t}\n\n\tws.Add(iter.WatchCh())\n\treturn iter, nil\n}\n","sourceCodeStart":1,"sourceCodeEnd":34,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/nomad/state/state_store_variables.go#L1-L34","documentation":"errVarAlreadyLocked is the sentinel returned when an ACL lock operation (VarLock/VarLockAcquire RPC) attempts to acquire a lock on a variable that already holds one. Nomad variables support advisory locking; a variable can only be locked by one holder at a time, so the second acquirer gets a 400 Bad Request wrapping this message.","triggerScenarios":"nomad var lock (VarLock RPC) on a path whose variable already has a lock held by another session/instance without the lock being released first; two workers racing to lock the same variable path; a crashed client whose lock was never released.","commonSituations":"Leader-election implementations built on nomad var lock where the old leader hasn't released; duplicate scheduler instances running concurrently; leaked locks from killed processes.","solutions":["Retry with backoff — in Nomad's lock semantics a failed acquire simply loses the election round; poll until the lock is released.","Check the lock holder via nomad var get <path> and coordinate release with the holder.","If the holder is dead, have an operator release it (nomad var unlock or VarLockRelease) and fix the client to release locks on shutdown.","Ensure only one process per workload attempts the lock (proper singleton deployment)."],"exampleFix":"// before: fails hard on contention\n_, _, err := client.Variables().Lock(path, writeOpts)\nif err != nil { return err }\n// after: retry loop for lock contention\nfor {\n    _, _, err := client.Variables().Lock(path, writeOpts)\n    if err == nil { break }\n    if strings.Contains(err.Error(), \"already holds a lock\") {\n        time.Sleep(2 * time.Second); continue\n    }\n    return err\n}","handlingStrategy":"retry","validationCode":"// inspect the variable's lock state before attempting acquisition\nv, _, err := client.Variables().Read(path, nil)\nif err == nil && v.Items[\"lock\"] != \"\" {\n    return fmt.Errorf(\"%s is locked; retry later\", path)\n}","typeGuard":null,"tryCatchPattern":"_, _, err := client.Variables().Lock(path, writeOpts)\nif err != nil && strings.Contains(err.Error(), \"already holds a lock\") {\n    time.Sleep(backoff)\n    return retryLock(path) // lose this election round gracefully\n}","preventionTips":["Design lock users as retry loops (leader election), not fail-fast.","Always release locks via defer on shutdown/panic.","Ensure single-instance deployment for lock-holding workloads.","Add lock TTL/heartbeat logic so crashed holders' locks get reclaimed."],"tags":["nomad","variables","locking","concurrency"],"backgroundTag":"lock-contention","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}