hashicorp/nomad · error

Emitting node events failed: %v

Error message

Emitting node events failed: %v

What it means

When the client flushes batched node events to the server via the Node.EmitEvents RPC, any RPC failure (connection refused, timeout, server not leader, auth rejection) is wrapped in this message. Node events (drain messages, custom events) are then retried by the batch watcher, so the error is often transient.

Source

Thrown at client/client.go:2040

// submitNodeEvents is used to submit a client-side node event. Examples of
// these kinds of events include when a driver moves from healthy to unhealthy
// (and vice versa)
func (c *Client) submitNodeEvents(events []*structs.NodeEvent) error {
	nodeID := c.NodeID()
	nodeEvents := map[string][]*structs.NodeEvent{
		nodeID: events,
	}
	req := structs.EmitNodeEventsRequest{
		NodeEvents: nodeEvents,
		WriteRequest: structs.WriteRequest{
			Region:    c.Region(),
			AuthToken: c.nodeAuthToken(),
		},
	}
	var resp structs.EmitNodeEventsResponse
	if err := c.RPC("Node.EmitEvents", &req, &resp); err != nil {
		return fmt.Errorf("Emitting node events failed: %v", err)
	}
	return nil
}

// watchNodeEvents is a handler which receives node events and on a interval
// and submits them in batch format to the server
func (c *Client) watchNodeEvents() {
	// batchEvents stores events that have yet to be published
	var batchEvents []*structs.NodeEvent

	timer := stoppedTimer()
	defer timer.Stop()

	for {
		select {
		case event := <-c.triggerEmitNodeEvent:
			if l := len(batchEvents); l <= structs.MaxRetainedNodeEvents {
				batchEvents = append(batchEvents, event)

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Check server availability and client logs for discovery/RPC errors (retry_join, Consul catalog)
2. Verify the client's ACL token/policy allows node:write for event emission
3. Validate TLS/CA configuration on both client and servers (mismatch fails all RPCs)
4. Ensure batching intervals aren't producing oversized requests on flaky links; restart the client to reset the event buffer if stuck

Example fix

# before: no servers reachable
client {
  servers = ["10.0.0.99:4647"]  # decommissioned server
}
// after
client {
  servers = ["10.0.0.1:4647", "10.0.0.2:4647", "10.0.0.3:4647"]
}

// before relying on event emission, verify RPC reachability
for _, s := range cfg.Servers {
	if err := probeRPC(s, 4647); err != nil {
		logger.Warn("server unreachable for node events", "server", s, "err", err)
	}
}

if err := emitNodeEvents(events); err != nil {
	logger.Warn("node event emit failed, will retry", "err", err)
	time.Sleep(emitRetryInterval) // events are re-batched; transient errors self-heal
}

When it happens

Trigger: c.RPC("Node.EmitEvents", ...) fails: no reachable server, servers down during maintenance, TLS/mTLS mismatch, ACL token lacking node events permission, or request timeout on a large event batch.

Common situations: Server restart/leader election while client emits events; network partition; misconfigured client server addresses; ACL policies changed to deny node write; clock/TLS issues after cert rotation.

Related errors

• failed to update status: %v
• failed to handle node update response: %w
• no servers
• ack.Error
• rcp.accept_backlog interval must be greater than zero

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/b82990d31a808286. Report an issue: GitHub.