{"record":{"id":"b82990d31a808286","repo":"hashicorp/nomad","slug":"emitting-node-events-failed-v","errorCode":null,"errorMessage":"Emitting node events failed: %v","messagePattern":"Emitting node events failed: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"client/client.go","lineNumber":2040,"sourceCode":"\n// submitNodeEvents is used to submit a client-side node event. Examples of\n// these kinds of events include when a driver moves from healthy to unhealthy\n// (and vice versa)\nfunc (c *Client) submitNodeEvents(events []*structs.NodeEvent) error {\n\tnodeID := c.NodeID()\n\tnodeEvents := map[string][]*structs.NodeEvent{\n\t\tnodeID: events,\n\t}\n\treq := structs.EmitNodeEventsRequest{\n\t\tNodeEvents: nodeEvents,\n\t\tWriteRequest: structs.WriteRequest{\n\t\t\tRegion:    c.Region(),\n\t\t\tAuthToken: c.nodeAuthToken(),\n\t\t},\n\t}\n\tvar resp structs.EmitNodeEventsResponse\n\tif err := c.RPC(\"Node.EmitEvents\", &req, &resp); err != nil {\n\t\treturn fmt.Errorf(\"Emitting node events failed: %v\", err)\n\t}\n\treturn nil\n}\n\n// watchNodeEvents is a handler which receives node events and on a interval\n// and submits them in batch format to the server\nfunc (c *Client) watchNodeEvents() {\n\t// batchEvents stores events that have yet to be published\n\tvar batchEvents []*structs.NodeEvent\n\n\ttimer := stoppedTimer()\n\tdefer timer.Stop()\n\n\tfor {\n\t\tselect {\n\t\tcase event := <-c.triggerEmitNodeEvent:\n\t\t\tif l := len(batchEvents); l <= structs.MaxRetainedNodeEvents {\n\t\t\t\tbatchEvents = append(batchEvents, event)","sourceCodeStart":2022,"sourceCodeEnd":2058,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/client/client.go#L2022-L2058","documentation":"When the client flushes batched node events to the server via the Node.EmitEvents RPC, any RPC failure (connection refused, timeout, server not leader, auth rejection) is wrapped in this message. Node events (drain messages, custom events) are then retried by the batch watcher, so the error is often transient.","triggerScenarios":"c.RPC(\"Node.EmitEvents\", ...) fails: no reachable server, servers down during maintenance, TLS/mTLS mismatch, ACL token lacking node events permission, or request timeout on a large event batch.","commonSituations":"Server restart/leader election while client emits events; network partition; misconfigured client server addresses; ACL policies changed to deny node write; clock/TLS issues after cert rotation.","solutions":["Check server availability and client logs for discovery/RPC errors (retry_join, Consul catalog)","Verify the client's ACL token/policy allows node:write for event emission","Validate TLS/CA configuration on both client and servers (mismatch fails all RPCs)","Ensure batching intervals aren't producing oversized requests on flaky links; restart the client to reset the event buffer if stuck"],"exampleFix":"# before: no servers reachable\nclient {\n  servers = [\"10.0.0.99:4647\"]  # decommissioned server\n}\n// after\nclient {\n  servers = [\"10.0.0.1:4647\", \"10.0.0.2:4647\", \"10.0.0.3:4647\"]\n}","handlingStrategy":"retry","validationCode":"// before relying on event emission, verify RPC reachability\nfor _, s := range cfg.Servers {\n\tif err := probeRPC(s, 4647); err != nil {\n\t\tlogger.Warn(\"server unreachable for node events\", \"server\", s, \"err\", err)\n\t}\n}","typeGuard":null,"tryCatchPattern":"if err := emitNodeEvents(events); err != nil {\n\tlogger.Warn(\"node event emit failed, will retry\", \"err\", err)\n\ttime.Sleep(emitRetryInterval) // events are re-batched; transient errors self-heal\n}","preventionTips":["Provide multiple server addresses / Consul discovery for failover","Keep mTLS certificates rotated before expiry","Grant the client's ACL token node:write","Avoid long server maintenance windows without draining clients"],"tags":["nomad-client","rpc","node-events"],"backgroundTag":"rpc-request-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}