hashicorp/nomad · error

error saving client identity: %w

Error message

error saving client identity: %w

What it means

When the server returns a renewed SignedIdentity in the heartbeat response, the client must persist it to the state DB before setting it in memory. If stateDB.PutNodeIdentity fails, the identity is intentionally NOT applied in memory (to avoid memory/disk divergence) and this error is returned; the client retries on the next heartbeat.

Source

Thrown at client/client.go:2329

	c.EnterpriseClient.SetFeatures(resp.Features)
	return nil
}

func (c *Client) handleNodeUpdateResponse(resp structs.NodeUpdateResponse) error {
	// Update the number of nodes in the cluster so we can adjust our server
	// rebalance rate.
	c.servers.SetNumNodes(resp.NumNodes)

	// If the response includes a new identity, set it and save it to the state
	// DB.
	//
	// In the unlikely event that we cannot write the identity to the state DB,
	// we do not want to set the client identity token. That would mean the
	// client memory state and persistent state DB are out of sync. Instead, we
	// return an error and wait until the next heartbeat to try again.
	if resp.SignedIdentity != nil {
		if err := c.stateDB.PutNodeIdentity(*resp.SignedIdentity); err != nil {
			return fmt.Errorf("error saving client identity: %w", err)
		}
		c.setNodeIdentityToken(*resp.SignedIdentity)

		// If the operator forced this renewal, reset the flag so that we don't
		// keep renewing the identity on every heartbeat.
		c.identityForceRenewal.Store(false)
	}

	// Convert []*NodeServerInfo to []*servers.Server
	nomadServers := make([]*servers.Server, 0, len(resp.Servers))
	for _, s := range resp.Servers {
		addr, err := resolveServer(s.RPCAdvertiseAddr)
		if err != nil {
			c.logger.Warn("ignoring invalid server", "error", err, "server", s.RPCAdvertiseAddr)
			continue
		}
		e := &servers.Server{Addr: addr}
		nomadServers = append(nomadServers, e)

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Read the wrapped error to identify the storage cause (enospc, permission denied, bolt corruption)
2. Free disk space or fix data_dir permissions so the state DB is writable
3. Restart the client; identity renewal is retried on the next heartbeat so the error can self-heal once storage is fixed
4. If state.db is corrupt, stop Nomad, back up and remove it, then restart (node re-registers)

Example fix

# before
$ df -h /var/lib/nomad
/dev/sda1  100%  used
// after
sudo journalctl --vacuum-size=100M
sudo systemctl restart nomad

// ensure state DB is writable before heartbeats expect identity writes
if err := probeWrite(cfg.DataDir); err != nil {
	return fmt.Errorf("identity persistence will fail: data_dir not writable: %w", err)
}
if freeDisk(cfg.DataDir) < minFreeBytes {
	return fmt.Errorf("insufficient disk for identity persistence")
}

if err := stateDB.PutNodeIdentity(identity); err != nil {
	logger.Error("identity not persisted; memory token NOT applied", "err", err)
	// do not set in-memory token; retry on next heartbeat
	return retryOnNextHeartbeat()
}

When it happens

Trigger: resp.SignedIdentity is non-nil but c.stateDB.PutNodeIdentity fails — corrupt/locked bolt DB, disk full, read-only filesystem, or I/O error on data_dir during a heartbeat with identity (re)issuance.

Common situations: Disk full on the client node; state.db corrupted after a crash; permissions on data_dir changed (e.g. by config management); first identity issuance right after client bootstrap on a node with failing storage.

Related errors

• node ID setup failed: %v
• error reading dynamic node metadata: %w
• error syncing dynamic node metadata: %w
• failed to remove alloc dir %q: %w
• Failed to make the alloc directory %v: %w

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/80495631ed6329f4. Report an issue: GitHub.