{"record":{"id":"f337a80c3cb72c76","repo":"hashicorp/nomad","slug":"failed-to-reset-heartbeat-since-server-is-not-lead","errorCode":null,"errorMessage":"failed to reset heartbeat since server is not leader","messagePattern":"failed to reset heartbeat since server is not leader","errorType":"error_code","errorClass":"heartbeatNotLeaderErr","httpStatus":null,"severity":"warning","filePath":"nomad/heartbeat.go","lineNumber":32,"sourceCode":"\n\t\"github.com/hashicorp/nomad/helper\"\n\t\"github.com/hashicorp/nomad/nomad/structs\"\n)\n\nconst (\n\t// heartbeatNotLeader is the error string returned when the heartbeat request\n\t// couldn't be completed since the server is not the leader.\n\theartbeatNotLeader = \"failed to reset heartbeat since server is not leader\"\n\n\t// NodeHeartbeatEventMissed is the event used when the Nodes heartbeat is\n\t// missed.\n\tNodeHeartbeatEventMissed = \"Node heartbeat missed\"\n)\n\nvar (\n\t// heartbeatNotLeaderErr is the error returned when the heartbeat request\n\t// couldn't be completed since the server is not the leader.\n\theartbeatNotLeaderErr = errors.New(heartbeatNotLeader)\n)\n\n// nodeHeartbeater is used to track expiration times of node heartbeats. If it\n// detects an expired node, the node status is updated to be 'down'.\ntype nodeHeartbeater struct {\n\tsrv    *Server\n\tlogger log.Logger\n\n\t// heartbeatTimers track the expiration time of each heartbeat that has\n\t// a TTL. On expiration, the node status is updated to be 'down'.\n\theartbeatTimers     map[string]*time.Timer\n\theartbeatTimersLock sync.Mutex\n}\n\n// newNodeHeartbeater returns a new node heartbeater used to detect and act on\n// failed node heartbeats.\nfunc newNodeHeartbeater(s *Server) *nodeHeartbeater {\n\treturn &nodeHeartbeater{","sourceCodeStart":14,"sourceCodeEnd":50,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/nomad/heartbeat.go#L14-L50","documentation":"heartbeatNotLeaderErr is returned by nodeHeartbeater.resetHeartbeatTimer when the RPC handling a node heartbeat lands on a server that is not the current Raft leader. Heartbeat TTL resets require leadership; the server logs a debug message and returns this error so the client can retry against the leader.","triggerScenarios":"A Nomad client sends Node.UpdateStatus (heartbeat) and the request is served (unforwarded or during a leadership transition) by a non-leader server — typically right after an election or when RPC forwarding is broken.","commonSituations":"Cluster during leader election/failed-over window; client connected directly to a follower with misconfigured servers list; network partition where clients reach followers but forwarding to the leader fails.","solutions":["Retry — clients typically retry and the new leader stabilizes; heartbeat TTLs tolerate short gaps","Ensure clients' 'servers' config lists all servers so they find the leader (check 'nomad node status' works)","Verify RPC forwarding between servers (ports 4646/4647 open, no firewall) so followers forward to the leader","Check 'nomad server members' and leader stability; fix flapping elections (Raft health, disk latency)"],"exampleFix":"// before: client pinned to a follower\nservers = [\"10.0.0.2:4647\"]  # follower only\n// after: list all servers so client discovers the leader\nservers = [\"10.0.0.1:4647\", \"10.0.0.2:4647\", \"10.0.0.3:4647\"]","handlingStrategy":"retry","validationCode":"// shell: only send heartbeats/update-status when the target is the leader\nleader=$(nomad server members | grep leader | awk '{print $2}')\n[ -n \"$leader\" ] || { echo 'no leader — retry later'; exit 1; }","typeGuard":null,"tryCatchPattern":"// Go\nttl, err := hb.resetHeartbeatTimer(id)\nif errors.Is(err, heartbeatNotLeaderErr) {\n    // transient: forward to leader or retry after backoff\n    return retryAfterBackoff(id)\n}","preventionTips":["List all servers in clients' servers config so they find the leader","Keep Raft stable: fast disks, quorum, monitor leader flapping","Ensure RPC forwarding ports (4646/4647) are open between all servers","Treat this error as transient with bounded retry + backoff"],"tags":["nomad","raft","leader-election","heartbeat"],"backgroundTag":"not-leader","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}