{"record":{"id":"a59d1132b09ed7e0","repo":"hashicorp/nomad","slug":"failed-lock-release-s","errorCode":null,"errorMessage":"failed lock release: %s","messagePattern":"failed lock release: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"nomad/state/state_store_variables.go","lineNumber":565,"sourceCode":"\t\t\tVariableMetadata: structs.VariableMetadata{\n\t\t\t\tNamespace: sv.Namespace,\n\t\t\t\tPath:      sv.Path,\n\t\t\t\tLock:      &structs.VariableLock{},\n\t\t\t},\n\t\t}\n\t\treturn req.ConflictResponse(idx, zeroVal)\n\t}\n\n\t// Avoid overwriting the variable data when releasing the lock, to prevent\n\t// a delay release to remove customer data.\n\n\tupdated := sv.Copy()\n\tupdated.Lock = nil\n\tupdated.ModifyIndex = idx\n\n\terr = s.updateVarsAndIndexTxn(tx, idx, &updated)\n\tif err != nil {\n\t\treq.ErrorResponse(idx, fmt.Errorf(\"failed lock release: %s\", err))\n\t}\n\n\tif err := tx.Commit(); err != nil {\n\t\treturn req.ErrorResponse(idx, err)\n\t}\n\n\treturn req.SuccessResponse(idx, &updated.VariableMetadata)\n}\n\nfunc (s *StateStore) updateVarsAndIndexTxn(tx WriteTxn, idx uint64, sv *structs.VariableEncrypted) error {\n\tif err := tx.Insert(TableVariables, sv); err != nil {\n\t\treturn fmt.Errorf(\"failed inserting variable: %w\", err)\n\t}\n\n\tif err := tx.Insert(tableIndex,\n\t\t&IndexEntry{TableVariables, idx}); err != nil {\n\t\treturn fmt.Errorf(\"failed updating variable index: %w\", err)\n\t}","sourceCodeStart":547,"sourceCodeEnd":583,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/nomad/state/state_store_variables.go#L547-L583","documentation":"VarLockRelease found the variable, copied it, cleared the Lock and bumped ModifyIndex, then called updateVarsAndIndexTxn to persist. This error wraps a failure of that persistence step (insert into variables table or index update inside the same write txn). The variable remains locked because the transaction was aborted (note the source also drops the return value of updateVarsAndIndexTxn, leaving the defer'd tx.Abort to roll back).","triggerScenarios":"tx.Insert into TableVariables or tableIndex fails inside updateVarsAndIndexTxn while applying OpReleaseLock in VarLockRelease; called from applyVariableOperation during FSM apply of a variable lock-release operation.","commonSituations":"Disk-full or IO errors on the server's data directory corrupting BoltDB writes; allocator/txn panics; multiple servers with divergent Raft state after an unclean shutdown.","solutions":["Inspect the wrapped cause for 'failed inserting variable' or 'failed updating variable index' details and check server logs at the same idx.","Free disk space / fix filesystem errors on the Nomad data_dir volume.","Restore state from a good snapshot if BoltDB is corrupt (nomad operator snapshot restore).","Retry the lock release after the store is healthy; the aborted txn guarantees no partial write."],"exampleFix":"// before (library code even drops the error return)\nerr = s.updateVarsAndIndexTxn(tx, idx, &updated)\nif err != nil { req.ErrorResponse(idx, fmt.Errorf(\"failed lock release: %s\", err)) }\n// after\nif err := s.updateVarsAndIndexTxn(tx, idx, &updated); err != nil {\n  return req.ErrorResponse(idx, fmt.Errorf(\"failed lock release: %w\", err))\n}","handlingStrategy":"retry","validationCode":null,"typeGuard":"func isLockReleaseWriteFailure(err error) bool { return err != nil && strings.Contains(err.Error(), \"failed lock release\") }","tryCatchPattern":"err := s.state.VarLockRelease(allocID, path, ns)\nif isLockReleaseWriteFailure(err) {\n  // txn aborted atomically; safe to retry after storage is healthy\n  backoff.Retry(func() error { return releaseLock(allocID, path, ns) }, backoffCfg)\n}","preventionTips":["Ensure adequate free space on the Nomad data_dir volume.","Use clean shutdowns; avoid killing servers during Raft writes.","Verify store health with snapshot inspect after unclean shutdowns."],"tags":["nomad","state-store","raft","persistence"],"backgroundTag":"state-store-write-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}