What causes "consul lock was lost" in hashicorp/terraform?

Verify Consul agent health and network connectivity; the lock loss is almost always infrastructural.

consul lock was lost

Error message

consul lock was lost

What it means

lostLockErr (internal/backend/remote-state/consul/client.go:40) is returned by unlock() (line 602) when, during unlock, the select on c.lockCh fires — meaning the Consul lock session was already lost/expired and the lock monitor goroutine previously reported '[ERROR] lost consul lock'. The client cannot cleanly release a lock it no longer holds; a background goroutine attempts reacquisition while CAS in Put guards against state corruption.

Source

Thrown at internal/backend/remote-state/consul/client.go:40

	"github.com/hashicorp/terraform/internal/states/statemgr"
	"github.com/hashicorp/terraform/internal/tfdiags"
)

const (
	lockSuffix     = "/.lock"
	lockInfoSuffix = "/.lockinfo"

	// The Session TTL associated with this lock.
	lockSessionTTL = "15s"

	// the delay time from when a session is lost to when the
	// lock is released by the server
	lockDelay = 5 * time.Second
	// interval between attempts to reacquire a lost lock
	lockReacquireInterval = 2 * time.Second
)

var lostLockErr = errors.New("consul lock was lost")

// RemoteClient is a remote client that stores data in Consul.
type RemoteClient struct {
	Client *consulapi.Client
	Path   string
	GZip   bool

	mu sync.Mutex
	// lockState is true if we're using locks
	lockState bool

	// The index of the last state we wrote.
	// If this is > 0, Put will perform a CAS to ensure that the state wasn't
	// changed during the operation. This is important even with locks, because
	// if the client loses the lock for some reason, then reacquires it, we
	// need to make sure that the state was not modified.
	modifyIndex uint64

View on GitHub (pinned to c9def3e214)

Solutions

Verify Consul agent health and network connectivity; the lock loss is almost always infrastructural.
Re-run the operation once Consul is stable — Put uses CAS (modifyIndex) to detect concurrent modification and avoid silent corruption.
If a stale lock entry remains, use 'terraform force-unlock' after confirming no other run is active.
Investigate session TTL / LockDelay tuning or consul agent resource pressure if it recurs.

Defensive patterns

Strategy: retry

Try / catch

// lostLockErr indicates infrastructural loss; do not treat as a normal unlock error.
if err := client.Unlock(id); err != nil {
    if errors.Is(err, lostLockErr) {
        log.Println("[WARN] consul lock was lost; relying on CAS for safety")
        // Optionally trigger force-unlock after confirming no active run.
    } else {
        return err
    }
}

Prevention

Monitor Consul agent health and network latency to the agent.
Keep terraform process responsive (avoid heavy GC/swap) so session renewal (TTL/2) is timely.
After a lost-lock event, rely on CAS (modifyIndex) and verify state integrity before proceeding.
Use 'terraform force-unlock' only after confirming no other run holds the lock.

When it happens

Trigger: Consul session TTL (15s) expired because the client could not renew it (network partition, Consul agent down, heavy GC pause); the lock monitor exhausted MonitorRetries and the session was invalidated. Unlock is then called at operation end and detects the lost lock.

Common situations: Network partition or Consul agent outage during a long apply; resource-constrained terraform process that can't renew the session in time; Consul cluster maintenance/failover mid-operation; aggressive firewall/load-balancer dropping the long-lived lock monitor connection.

Related errors

AI-assisted analysis of hashicorp/terraform@c9def3e214 (2026-08-07). Data as JSON: /api/errors/19a470b199708ded. Report an issue: GitHub.