{"record":{"id":"4eafa99f2b222789","repo":"hashicorp/nomad","slug":"raft-apply-failed-w","errorCode":null,"errorMessage":"raft apply failed: %w","messagePattern":"raft apply failed: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"nomad/variables_endpoint.go","lineNumber":141,"sourceCode":"\t\t\tVariableMetadata: structs.VariableMetadata{\n\t\t\t\tNamespace:   args.Var.Namespace,\n\t\t\t\tPath:        args.Var.Path,\n\t\t\t\tModifyIndex: args.Var.ModifyIndex,\n\t\t\t},\n\t\t}\n\t}\n\n\t// Make a SVEArgs\n\tsveArgs := structs.VarApplyStateRequest{\n\t\tOp:           args.Op,\n\t\tVar:          ev,\n\t\tWriteRequest: args.WriteRequest,\n\t}\n\n\t// Apply the update.\n\to, index, err := sv.srv.raftApply(structs.VarApplyStateRequestType, sveArgs)\n\tif err != nil {\n\t\treturn fmt.Errorf(\"raft apply failed: %w\", err)\n\t}\n\n\tout, _ := o.(*structs.VarApplyStateResponse)\n\n\t// The return value depends on the operation results and the callers permissions\n\tr, err := sv.makeVariablesApplyResponse(args, out, aclObj)\n\tif err != nil {\n\t\treturn err\n\t}\n\n\t*reply = *r\n\treply.Index = index\n\n\tif out.IsOk() {\n\t\tswitch args.Op {\n\t\tcase structs.VarOpLockAcquire:\n\t\t\tsv.timers.CreateVariableLockTTLTimer(ev.Copy())\n\t\tcase structs.VarOpLockRelease:","sourceCodeStart":123,"sourceCodeEnd":159,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/nomad/variables_endpoint.go#L123-L159","documentation":"After validation and encryption, Apply commits the variable through Raft (VarApplyStateRequestType). Any error from raftApply — no leader, lost quorum, FSM apply rejection — is wrapped as 'raft apply failed: %w'. The variable write did not commit.","triggerScenarios":"nomad var put / SDK Apply during leader election, quorum loss, Raft store errors, or when the FSM rejects the apply (e.g. CAS mismatch handled separately, but transport/commit failures surface here).","commonSituations":"Writing variables while servers are down; network partition between client and leader; Raft disk issues on the leader; retry storms during upgrades.","solutions":["Check cluster leadership (`nomad operator api /v1/status/leader`) and retry after a leader exists","Restore quorum or fix failing Raft servers; inspect server logs for the root error","Retry the apply with backoff — writes are safe to reissue","For CAS ops, re-fetch the current ModifyIndex and retry with the fresh index"],"exampleFix":"// before\n_, err := vars.Apply(req) // fails during election\n\n// after\nerr := retry.Do(func() error { _, err := vars.Apply(req); return err }, retry.Delay(time.Second))","handlingStrategy":"retry","validationCode":"leader, _ := client.Status().Leader(ctx)\nif leader == \"\" {\n    return errors.New(\"no leader: defer variable writes until cluster is stable\")\n}","typeGuard":null,"tryCatchPattern":"err := retry.Do(func() error {\n    _, err := client.Variables().Apply(req, nil)\n    if err != nil && strings.Contains(err.Error(), \"raft apply failed\") {\n        return err // retryable\n    }\n    return nil\n}, retry.Attempts(5), retry.DelayType(retry.BackOffDelay))","preventionTips":["Retry writes with backoff during elections/upgrades","For CAS ops, refresh ModifyIndex before retrying","Alert on quorum loss before running write-heavy jobs"],"tags":["nomad","variables","raft","leader-election"],"backgroundTag":"raft-no-leader","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}