What causes "variable already holds a lock" in hashicorp/nomad?

Retry with backoff — in Nomad's lock semantics a failed acquire simply loses the election round; poll until the lock is released.

hashicorp/nomad · warning

variable already holds a lock

Error message

variable already holds a lock

What it means

errVarAlreadyLocked is the sentinel returned when an ACL lock operation (VarLock/VarLockAcquire RPC) attempts to acquire a lock on a variable that already holds one. Nomad variables support advisory locking; a variable can only be locked by one holder at a time, so the second acquirer gets a 400 Bad Request wrapping this message.

Source

Thrown at nomad/state/state_store_variables.go:16

// Copyright IBM Corp. 2015, 2026
// SPDX-License-Identifier: BUSL-1.1

package state

import (
	"errors"
	"fmt"
	"math"

	"github.com/hashicorp/go-memdb"
	"github.com/hashicorp/nomad/nomad/structs"
)

var (
	errVarAlreadyLocked = errors.New("variable already holds a lock")
	errVarNotFound      = errors.New("variable doesn't exist")
	errLockNotFound     = errors.New("variable doesn't hold a lock")
)

// Variables queries all the variables and is used only for
// snapshot/restore and key rotation
func (s *StateStore) Variables(ws memdb.WatchSet) (memdb.ResultIterator, error) {
	txn := s.db.ReadTxn()

	iter, err := txn.Get(TableVariables, indexID)
	if err != nil {
		return nil, err
	}

	ws.Add(iter.WatchCh())
	return iter, nil
}

View on GitHub (pinned to 482b49bf1a)

Solutions

Retry with backoff — in Nomad's lock semantics a failed acquire simply loses the election round; poll until the lock is released.
Check the lock holder via nomad var get <path> and coordinate release with the holder.
If the holder is dead, have an operator release it (nomad var unlock or VarLockRelease) and fix the client to release locks on shutdown.
Ensure only one process per workload attempts the lock (proper singleton deployment).

Example fix

// before: fails hard on contention
_, _, err := client.Variables().Lock(path, writeOpts)
if err != nil { return err }
// after: retry loop for lock contention
for {
    _, _, err := client.Variables().Lock(path, writeOpts)
    if err == nil { break }
    if strings.Contains(err.Error(), "already holds a lock") {
        time.Sleep(2 * time.Second); continue
    }
    return err
}

Defensive patterns

Strategy: retry

Validate before calling

// inspect the variable's lock state before attempting acquisition
v, _, err := client.Variables().Read(path, nil)
if err == nil && v.Items["lock"] != "" {
    return fmt.Errorf("%s is locked; retry later", path)
}

Try / catch

_, _, err := client.Variables().Lock(path, writeOpts)
if err != nil && strings.Contains(err.Error(), "already holds a lock") {
    time.Sleep(backoff)
    return retryLock(path) // lose this election round gracefully
}

Prevention

Design lock users as retry loops (leader election), not fail-fast.
Always release locks via defer on shutdown/panic.
Ensure single-instance deployment for lock-holding workloads.
Add lock TTL/heartbeat logic so crashed holders' locks get reclaimed.

When it happens

Trigger: nomad var lock (VarLock RPC) on a path whose variable already has a lock held by another session/instance without the lock being released first; two workers racing to lock the same variable path; a crashed client whose lock was never released.

Common situations: Leader-election implementations built on nomad var lock where the old leader hasn't released; duplicate scheduler instances running concurrently; leaked locks from killed processes.

Related errors

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/792037ceb6174f5d. Report an issue: GitHub.