hashicorp/nomad · warning · heartbeatNotLeaderErr

failed to reset heartbeat since server is not leader

Error message

failed to reset heartbeat since server is not leader

What it means

heartbeatNotLeaderErr is returned by nodeHeartbeater.resetHeartbeatTimer when the RPC handling a node heartbeat lands on a server that is not the current Raft leader. Heartbeat TTL resets require leadership; the server logs a debug message and returns this error so the client can retry against the leader.

Source

Thrown at nomad/heartbeat.go:32

	"github.com/hashicorp/nomad/helper"
	"github.com/hashicorp/nomad/nomad/structs"
)

const (
	// heartbeatNotLeader is the error string returned when the heartbeat request
	// couldn't be completed since the server is not the leader.
	heartbeatNotLeader = "failed to reset heartbeat since server is not leader"

	// NodeHeartbeatEventMissed is the event used when the Nodes heartbeat is
	// missed.
	NodeHeartbeatEventMissed = "Node heartbeat missed"
)

var (
	// heartbeatNotLeaderErr is the error returned when the heartbeat request
	// couldn't be completed since the server is not the leader.
	heartbeatNotLeaderErr = errors.New(heartbeatNotLeader)
)

// nodeHeartbeater is used to track expiration times of node heartbeats. If it
// detects an expired node, the node status is updated to be 'down'.
type nodeHeartbeater struct {
	srv    *Server
	logger log.Logger

	// heartbeatTimers track the expiration time of each heartbeat that has
	// a TTL. On expiration, the node status is updated to be 'down'.
	heartbeatTimers     map[string]*time.Timer
	heartbeatTimersLock sync.Mutex
}

// newNodeHeartbeater returns a new node heartbeater used to detect and act on
// failed node heartbeats.
func newNodeHeartbeater(s *Server) *nodeHeartbeater {
	return &nodeHeartbeater{

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Retry — clients typically retry and the new leader stabilizes; heartbeat TTLs tolerate short gaps
2. Ensure clients' 'servers' config lists all servers so they find the leader (check 'nomad node status' works)
3. Verify RPC forwarding between servers (ports 4646/4647 open, no firewall) so followers forward to the leader
4. Check 'nomad server members' and leader stability; fix flapping elections (Raft health, disk latency)

Example fix

// before: client pinned to a follower
servers = ["10.0.0.2:4647"]  # follower only
// after: list all servers so client discovers the leader
servers = ["10.0.0.1:4647", "10.0.0.2:4647", "10.0.0.3:4647"]

// shell: only send heartbeats/update-status when the target is the leader
leader=$(nomad server members | grep leader | awk '{print $2}')
[ -n "$leader" ] || { echo 'no leader — retry later'; exit 1; }

// Go
ttl, err := hb.resetHeartbeatTimer(id)
if errors.Is(err, heartbeatNotLeaderErr) {
    // transient: forward to leader or retry after backoff
    return retryAfterBackoff(id)
}

When it happens

Trigger: A Nomad client sends Node.UpdateStatus (heartbeat) and the request is served (unforwarded or during a leadership transition) by a non-leader server — typically right after an election or when RPC forwarding is broken.

Common situations: Cluster during leader election/failed-over window; client connected directly to a follower with misconfigured servers list; network partition where clients reach followers but forwarding to the leader fails.

Related errors

• unsupported minimum common raft protocol version
• must provide peer id or address
• No cluster leader
• reconciliation of job summaries failed: %v
• raft apply failed: %w

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/f337a80c3cb72c76. Report an issue: GitHub.