{"record":{"id":"2e118ec7bd9aad1a","repo":"hashicorp/nomad","slug":"failed-to-update-status-v","errorCode":null,"errorMessage":"failed to update status: %v","messagePattern":"failed to update status: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"client/client.go","lineNumber":2253,"sourceCode":"\t\tNodeID: c.NodeID(),\n\t\tStatus: structs.NodeStatusReady,\n\t\tWriteRequest: structs.WriteRequest{\n\t\t\tRegion:    c.Region(),\n\t\t\tAuthToken: c.nodeAuthToken(),\n\t\t},\n\t}\n\n\t// Check if the client has been informed to force a renewal of its identity,\n\t// and set the flag in the request if so.\n\tif c.identityForceRenewal.Load() {\n\t\tc.logger.Debug(\"forcing identity renewal\")\n\t\treq.ForceIdentityRenewal = true\n\t}\n\n\tvar resp structs.NodeUpdateResponse\n\tif err := c.RPC(\"Node.UpdateStatus\", &req, &resp); err != nil {\n\t\tc.triggerDiscovery()\n\t\treturn fmt.Errorf(\"failed to update status: %v\", err)\n\t}\n\tendTime := time.Now()\n\n\tif len(resp.EvalIDs) != 0 {\n\t\tc.logger.Debug(\"evaluations triggered by node update\", \"num_evals\", len(resp.EvalIDs))\n\t}\n\n\t// Update the last heartbeat and the new TTL, capturing the old values\n\tc.heartbeatLock.Lock()\n\tlast := c.lastHeartbeat()\n\toldTTL := c.heartbeatTTL\n\thaveHeartbeated := c.haveHeartbeated\n\tc.heartbeatStop.setLastOk(endTime)\n\tc.heartbeatTTL = resp.HeartbeatTTL\n\tc.haveHeartbeated = true\n\tc.heartbeatLock.Unlock()\n\tc.logger.Trace(\"next heartbeat\", \"period\", resp.HeartbeatTTL)\n","sourceCodeStart":2235,"sourceCodeEnd":2271,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/client/client.go#L2235-L2271","documentation":"The client heartbeats its status to the servers via the Node.UpdateStatus RPC; on RPC failure it triggers discovery (to find new servers) and wraps the error in this message. Failing heartbeats eventually cause the server to mark the node as down and reschedule its allocations, so persistent occurrences are serious.","triggerScenarios":"c.RPC(\"Node.UpdateStatus\", ...) fails during heartbeat or initial registration: unreachable servers, connection refused, TLS handshake failure, RPC timeout on overloaded servers, or an auth token rejected by servers.","commonSituations":"Network partition between client and server subnet; servers rolled in a maintenance window; firewall blocking port 4647; mTLS certs expired; Consul used for discovery returning stale addresses.","solutions":["Read the wrapped error: connection refused/timeout → networking; x509 → TLS; permission denied → ACL","Verify servers are reachable from the client on port 4647 (telnet/nc) and that discovery (Consul/retry_join) returns live addresses","Fix TLS configuration or renew expired mTLS certificates on the agent","Check server health/leader stability; after servers return, the client's next heartbeat re-registers the node"],"exampleFix":"// before\n# firewall drops 4647\n// after\nsudo ufw allow 4647/tcp\nsudo systemctl restart nomad","handlingStrategy":"retry","validationCode":"// preflight connectivity check before agent start\nconn, err := net.DialTimeout(\"tcp\", serverAddr, 3*time.Second)\nif err != nil {\n\treturn fmt.Errorf(\"cannot reach server %s:4647 before heartbeat: %w\", serverAddr, err)\n}\nconn.Close()","typeGuard":null,"tryCatchPattern":"if err := updateStatus(); err != nil {\n\tlogger.Error(\"heartbeat failed\", \"err\", err)\n\ttriggerDiscovery() // rediscover servers, then retry with backoff\n\treturn retryHeartbeat()\n}","preventionTips":["Open port 4647 between client and server subnets in firewalls","Use Consul-based discovery instead of static stale addresses","Monitor heartbeat expiry / node 'down' events in Nomad and alert","Track TLS certificate expiry for mTLS deployments"],"tags":["nomad-client","heartbeat","rpc"],"backgroundTag":"heartbeat-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}