hashicorp/nomad · critical

node ID setup failed: %v

Error message

node ID setup failed: %v

What it means

Nomad's client wraps any failure from ensureNodeID, which generates or restores the node's ID and secret ID (typically via the state DB or server registration), with this message. Without a node ID the client cannot register or heartbeat with the servers, so setup aborts. The wrapped %v carries the underlying cause (state DB failure, RPC error, etc.).

Source

Thrown at client/client.go:1616

	return id, secret, nil
}

// setupNode is used to setup the initial node
func (c *Client) setupNode() error {
	c.configLock.Lock()
	defer c.configLock.Unlock()

	newConfig := c.config.Copy()
	node := newConfig.Node
	if node == nil {
		node = &structs.Node{}
		newConfig.Node = node
	}

	// Generate an ID and secret for the node
	id, secretID, err := ensureNodeID(newConfig)
	if err != nil {
		return fmt.Errorf("node ID setup failed: %v", err)
	}

	node.ID = id
	node.SecretID = secretID
	if node.Attributes == nil {
		node.Attributes = make(map[string]string)
	}
	if node.Links == nil {
		node.Links = make(map[string]string)
	}
	if node.Drivers == nil {
		node.Drivers = make(map[string]*structs.DriverInfo)
	}
	if node.CSIControllerPlugins == nil {
		node.CSIControllerPlugins = make(map[string]*structs.CSIInfo)
	}
	if node.CSINodePlugins == nil {
		node.CSINodePlugins = make(map[string]*structs.CSIInfo)

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Check the wrapped inner error message to identify whether it is a state DB or RPC failure
2. Verify the Nomad data_dir exists, is writable by the Nomad user, and has correct permissions
3. Stop the agent, back up, and remove/repair the client state DB (state.db) so the ID can be regenerated
4. Ensure servers are reachable (retry_join/consul discovery) if ID provisioning requires RPC

Example fix

// before
node {
  # no explicit id, relying on corrupt state.db
}
// after
# stop nomad, fix permissions, or set a stable ID
node {
  id = "fixed-node-id"
}

// before starting the client
if err := ensureDataDirWritable(cfg.DataDir); err != nil {
	return fmt.Errorf("data_dir not writable, node ID setup will fail: %w", err)
}
if cfg.NodeID == "" {
	// ensure state.db is intact or pin a stable node.id
}

id, secretID, err := ensureNodeID(cfg)
if err != nil {
	logger.Error("node ID setup failed", "err", err)
	return retry.WithBackoff(func() error { return ensureNodeID(cfg) })
}

When it happens

Trigger: ensureNodeID returns an error: the local state DB cannot read/write the stored node identity, or generating/renewing the node secret fails during client setup (client.NewClient / node setup path).

Common situations: Corrupt or unwritable Nomad data_dir; stale identity rows in the boltdb state store; server unreachable during initial ID provisioning; permissions issues on the data directory after running as a different user.

Related errors

• error reading dynamic node metadata: %w
• error syncing dynamic node metadata: %w
• error saving client identity: %w
• failed to remove alloc dir %q: %w
• Failed to make the alloc directory %v: %w

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/018c502159df8203. Report an issue: GitHub.