hashicorp/nomad · critical

failed to update status: %v

Error message

failed to update status: %v

What it means

The client heartbeats its status to the servers via the Node.UpdateStatus RPC; on RPC failure it triggers discovery (to find new servers) and wraps the error in this message. Failing heartbeats eventually cause the server to mark the node as down and reschedule its allocations, so persistent occurrences are serious.

Source

Thrown at client/client.go:2253

		NodeID: c.NodeID(),
		Status: structs.NodeStatusReady,
		WriteRequest: structs.WriteRequest{
			Region:    c.Region(),
			AuthToken: c.nodeAuthToken(),
		},
	}

	// Check if the client has been informed to force a renewal of its identity,
	// and set the flag in the request if so.
	if c.identityForceRenewal.Load() {
		c.logger.Debug("forcing identity renewal")
		req.ForceIdentityRenewal = true
	}

	var resp structs.NodeUpdateResponse
	if err := c.RPC("Node.UpdateStatus", &req, &resp); err != nil {
		c.triggerDiscovery()
		return fmt.Errorf("failed to update status: %v", err)
	}
	endTime := time.Now()

	if len(resp.EvalIDs) != 0 {
		c.logger.Debug("evaluations triggered by node update", "num_evals", len(resp.EvalIDs))
	}

	// Update the last heartbeat and the new TTL, capturing the old values
	c.heartbeatLock.Lock()
	last := c.lastHeartbeat()
	oldTTL := c.heartbeatTTL
	haveHeartbeated := c.haveHeartbeated
	c.heartbeatStop.setLastOk(endTime)
	c.heartbeatTTL = resp.HeartbeatTTL
	c.haveHeartbeated = true
	c.heartbeatLock.Unlock()
	c.logger.Trace("next heartbeat", "period", resp.HeartbeatTTL)

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Read the wrapped error: connection refused/timeout → networking; x509 → TLS; permission denied → ACL
2. Verify servers are reachable from the client on port 4647 (telnet/nc) and that discovery (Consul/retry_join) returns live addresses
3. Fix TLS configuration or renew expired mTLS certificates on the agent
4. Check server health/leader stability; after servers return, the client's next heartbeat re-registers the node

Example fix

// before
# firewall drops 4647
// after
sudo ufw allow 4647/tcp
sudo systemctl restart nomad

// preflight connectivity check before agent start
conn, err := net.DialTimeout("tcp", serverAddr, 3*time.Second)
if err != nil {
	return fmt.Errorf("cannot reach server %s:4647 before heartbeat: %w", serverAddr, err)
}
conn.Close()

if err := updateStatus(); err != nil {
	logger.Error("heartbeat failed", "err", err)
	triggerDiscovery() // rediscover servers, then retry with backoff
	return retryHeartbeat()
}

When it happens

Trigger: c.RPC("Node.UpdateStatus", ...) fails during heartbeat or initial registration: unreachable servers, connection refused, TLS handshake failure, RPC timeout on overloaded servers, or an auth token rejected by servers.

Common situations: Network partition between client and server subnet; servers rolled in a maintenance window; firewall blocking port 4647; mTLS certs expired; Consul used for discovery returning stale addresses.

Related errors

• failed to handle node update response: %w
• Emitting node events failed: %v
• no servers
• ack.Error
• rcp.accept_backlog interval must be greater than zero

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/2e118ec7bd9aad1a. Report an issue: GitHub.