{"record":{"id":"1a70be2cbb71f5ff","repo":"hashicorp/nomad","slug":"failed-inserting-variable-w","errorCode":null,"errorMessage":"failed inserting variable: %w","messagePattern":"failed inserting variable: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"nomad/state/state_store_variables.go","lineNumber":577,"sourceCode":"\tupdated := sv.Copy()\n\tupdated.Lock = nil\n\tupdated.ModifyIndex = idx\n\n\terr = s.updateVarsAndIndexTxn(tx, idx, &updated)\n\tif err != nil {\n\t\treq.ErrorResponse(idx, fmt.Errorf(\"failed lock release: %s\", err))\n\t}\n\n\tif err := tx.Commit(); err != nil {\n\t\treturn req.ErrorResponse(idx, err)\n\t}\n\n\treturn req.SuccessResponse(idx, &updated.VariableMetadata)\n}\n\nfunc (s *StateStore) updateVarsAndIndexTxn(tx WriteTxn, idx uint64, sv *structs.VariableEncrypted) error {\n\tif err := tx.Insert(TableVariables, sv); err != nil {\n\t\treturn fmt.Errorf(\"failed inserting variable: %w\", err)\n\t}\n\n\tif err := tx.Insert(tableIndex,\n\t\t&IndexEntry{TableVariables, idx}); err != nil {\n\t\treturn fmt.Errorf(\"failed updating variable index: %w\", err)\n\t}\n\treturn nil\n}\n\nfunc isLocked(lock *structs.VariableLock, req *structs.VarApplyStateRequest) bool {\n\tif lock != nil {\n\t\tif req.Var.VariableMetadata.Lock == nil ||\n\t\t\treq.Var.VariableMetadata.Lock.ID != lock.ID {\n\n\t\t\treturn true\n\t\t}\n\t}\n\treturn false","sourceCodeStart":559,"sourceCodeEnd":595,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/nomad/state/state_store_variables.go#L559-L595","documentation":"updateVarsAndIndexTxn inserts the VariableEncrypted object into the variables table within the active write transaction. This error wraps tx.Insert failing for the variable entry itself, aborting the whole update (variable + index) since it returns early. Callers like VarLockRelease and VarApply propagate it up as the RPC error response.","triggerScenarios":"tx.Insert(TableVariables, sv) returns an error during any variable write (create/update/lock release) applied to the state store.","commonSituations":"BoltDB write failures from disk-full, permission problems on data_dir, or memory pressure in the memdb layer on large variables; seen after unclean shutdowns leaving the store in a bad state.","solutions":["Read the wrapped %w cause; if it mentions BoltDB/IO, check data_dir disk space and permissions.","Restart the Nomad server process to rebuild the in-memory state from disk.","Restore from a valid snapshot if the store is corrupted.","Retry the variable write; the failed txn was fully rolled back so nothing is half-written."],"exampleFix":"// before\nif err := client.State().SetVariable(sv); err != nil { panic(err) }\n// after\nif err := client.State().SetVariable(sv); err != nil {\n  if strings.Contains(err.Error(), \"failed inserting variable\") {\n    logger.Error(\"variable write rejected by state store\", \"path\", sv.Path, \"err\", err)\n  }\n}","handlingStrategy":"retry","validationCode":null,"typeGuard":"func isVarInsertFailure(err error) bool { return err != nil && strings.Contains(err.Error(), \"failed inserting variable\") }","tryCatchPattern":"if err := applyVariable(op); err != nil {\n  if isVarInsertFailure(err) {\n    logger.Error(\"variable write rolled back\", \"path\", op.Var.Path, \"err\", err)\n    return retryWithBackoff(op)\n  }\n  return err\n}","preventionTips":["Watch BoltDB/IO errors in server logs; fix disk pressure early.","Keep variable payloads small to reduce memory pressure.","Restore from snapshot promptly if corruption is suspected."],"tags":["nomad","state-store","storage","variables"],"backgroundTag":"state-store-write-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}