{"record":{"id":"19a470b199708ded","repo":"hashicorp/terraform","slug":"consul-lock-was-lost","errorCode":null,"errorMessage":"consul lock was lost","messagePattern":"consul lock was lost","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"internal/backend/remote-state/consul/client.go","lineNumber":40,"sourceCode":"\t\"github.com/hashicorp/terraform/internal/states/statemgr\"\n\t\"github.com/hashicorp/terraform/internal/tfdiags\"\n)\n\nconst (\n\tlockSuffix     = \"/.lock\"\n\tlockInfoSuffix = \"/.lockinfo\"\n\n\t// The Session TTL associated with this lock.\n\tlockSessionTTL = \"15s\"\n\n\t// the delay time from when a session is lost to when the\n\t// lock is released by the server\n\tlockDelay = 5 * time.Second\n\t// interval between attempts to reacquire a lost lock\n\tlockReacquireInterval = 2 * time.Second\n)\n\nvar lostLockErr = errors.New(\"consul lock was lost\")\n\n// RemoteClient is a remote client that stores data in Consul.\ntype RemoteClient struct {\n\tClient *consulapi.Client\n\tPath   string\n\tGZip   bool\n\n\tmu sync.Mutex\n\t// lockState is true if we're using locks\n\tlockState bool\n\n\t// The index of the last state we wrote.\n\t// If this is > 0, Put will perform a CAS to ensure that the state wasn't\n\t// changed during the operation. This is important even with locks, because\n\t// if the client loses the lock for some reason, then reacquires it, we\n\t// need to make sure that the state was not modified.\n\tmodifyIndex uint64\n","sourceCodeStart":22,"sourceCodeEnd":58,"githubUrl":"https://github.com/hashicorp/terraform/blob/c9def3e214014c1188faabfc4a5bde5095139765/internal/backend/remote-state/consul/client.go#L22-L58","documentation":"lostLockErr (internal/backend/remote-state/consul/client.go:40) is returned by unlock() (line 602) when, during unlock, the select on c.lockCh fires — meaning the Consul lock session was already lost/expired and the lock monitor goroutine previously reported '[ERROR] lost consul lock'. The client cannot cleanly release a lock it no longer holds; a background goroutine attempts reacquisition while CAS in Put guards against state corruption.","triggerScenarios":"Consul session TTL (15s) expired because the client could not renew it (network partition, Consul agent down, heavy GC pause); the lock monitor exhausted MonitorRetries and the session was invalidated. Unlock is then called at operation end and detects the lost lock.","commonSituations":"Network partition or Consul agent outage during a long apply; resource-constrained terraform process that can't renew the session in time; Consul cluster maintenance/failover mid-operation; aggressive firewall/load-balancer dropping the long-lived lock monitor connection.","solutions":["Verify Consul agent health and network connectivity; the lock loss is almost always infrastructural.","Re-run the operation once Consul is stable — Put uses CAS (modifyIndex) to detect concurrent modification and avoid silent corruption.","If a stale lock entry remains, use 'terraform force-unlock' after confirming no other run is active.","Investigate session TTL / LockDelay tuning or consul agent resource pressure if it recurs."],"exampleFix":null,"handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"// lostLockErr indicates infrastructural loss; do not treat as a normal unlock error.\nif err := client.Unlock(id); err != nil {\n    if errors.Is(err, lostLockErr) {\n        log.Println(\"[WARN] consul lock was lost; relying on CAS for safety\")\n        // Optionally trigger force-unlock after confirming no active run.\n    } else {\n        return err\n    }\n}","preventionTips":["Monitor Consul agent health and network latency to the agent.","Keep terraform process responsive (avoid heavy GC/swap) so session renewal (TTL/2) is timely.","After a lost-lock event, rely on CAS (modifyIndex) and verify state integrity before proceeding.","Use 'terraform force-unlock' only after confirming no other run holds the lock."],"tags":["backend","consul","locking","distributed-lock","network"],"analyzedSha":"c9def3e214014c1188faabfc4a5bde5095139765","analyzedAt":"2026-08-07T15:39:49.278Z","schemaVersion":2},"datasetVersion":"2026-08-07T20:17:04.800Z"}