{"record":{"id":"caa47746ec3a3896","repo":"hashicorp/nomad","slug":"variable-quota-insert-failed-v","errorCode":null,"errorMessage":"variable quota insert failed: %v","messagePattern":"variable quota insert failed: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"nomad/state/state_store_variables.go","lineNumber":288,"sourceCode":"\n\tif quotaChange > 0 {\n\t\tquotaUsed.Size += quotaChange\n\t} else if quotaChange < 0 {\n\t\tquotaUsed.Size -= min(quotaUsed.Size, -quotaChange)\n\t}\n\n\terr = s.enforceVariablesQuota(idx, tx, sv.Namespace, quotaChange)\n\tif err != nil {\n\t\treturn req.ErrorResponse(idx, err)\n\t}\n\n\t// we check enforcement above even if there's no change because another\n\t// namespace may have used up quota to make this no longer valid, but we\n\t// only update the table if this namespace has changed\n\tif quotaChange != 0 {\n\t\tquotaUsed.ModifyIndex = idx\n\t\tif err := tx.Insert(TableVariablesQuotas, quotaUsed); err != nil {\n\t\t\treturn req.ErrorResponse(idx, fmt.Errorf(\"variable quota insert failed: %v\", err))\n\t\t}\n\t}\n\n\tif err := tx.Insert(tableIndex,\n\t\t&IndexEntry{TableVariables, idx}); err != nil {\n\t\treturn req.ErrorResponse(idx, fmt.Errorf(\"failed updating variable index: %s\", err))\n\t}\n\n\treturn req.SuccessResponse(idx, &sv.VariableMetadata)\n}\n\n// VarDelete is used to delete a single variable in the\n// the state store.\nfunc (s *StateStore) VarDelete(msgType structs.MessageType, idx uint64, req *structs.VarApplyStateRequest) *structs.VarApplyStateResponse {\n\ttx := s.db.WriteTxnMsgT(msgType, idx)\n\tdefer tx.Abort()\n\n\t// Perform the actual delete","sourceCodeStart":270,"sourceCodeEnd":306,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/nomad/state/state_store_variables.go#L270-L306","documentation":"This error is returned by varSetTxn when inserting/updating the namespace's variables quota usage row (TableVariablesQuotas) into the memdb state store fails. The transaction cannot record the new quota consumption, so the variable write is aborted and an ErrorResponse is produced. It wraps the underlying memdb insert error (e.g. transaction already aborted, invalid object).","triggerScenarios":"Calling VarSet, VarSetCAS (varSetCASTxn) or VarLockAcquire for a variable in a namespace where quotaChanged != 0 (quota tracked and size changed) and the tx.Insert(TableVariablesQuotas, quotaUsed) call fails — typically due to an aborted/invalid write txn, a corrupt or wrong-type quota row, or an internal memdb error.","commonSituations":"State store internals failing during heavy write load; Raft apply path applying a variable write while the memdb transaction is in a bad state; corrupted state store after snapshot restore; developer bugs introducing objects of unexpected type into TableVariablesQuotas.","solutions":["Inspect the wrapped error (%v) to identify the memdb failure cause","Retry the VarSet request — the write is transactional and safely aborted, so retrying is safe","Verify state store integrity (restore from a known-good snapshot if corruption is suspected)","Upgrade Nomad if the wrapped error points to a known memdb bug"],"exampleFix":"// before\nreturn req.ErrorResponse(idx, fmt.Errorf(\"variable quota insert failed: %v\", err))\n// after\n// add context for operators\nreturn req.ErrorResponse(idx, fmt.Errorf(\"variable quota insert failed (namespace=%s): %w\", sv.Namespace, err))","handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"resp, err := client.Variables().Create(var)\nif err != nil && strings.Contains(err.Error(), \"variable quota insert failed\") {\n    // transient state-store failure; retry with backoff\n    resp, err = backoff.Retry(func() (*vars.VariableWriteResponse, error) { return client.Variables().Create(var) })\n}","preventionTips":["Treat variable writes as retryable; implement client backoff","Keep state store healthy: monitor Raft and snapshot/restore procedures","Avoid custom code writing directly to TableVariablesQuotas"],"tags":["nomad","state-store","memdb","variables","quota"],"backgroundTag":"memdb-insert-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}