hashicorp/nomad · error

error reading dynamic node metadata: %w

Error message

error reading dynamic node metadata: %w

What it means

The client merges dynamic node metadata (tombstone-aware key/value metadata persisted in the local state DB) into the node during setup. If stateDB.GetNodeMeta fails to read that data, setup aborts with this wrapped error. It indicates local persistent state is unreadable, not a server-side problem.

Source

Thrown at client/client.go:1715

		node.Meta[envoy.DefaultConnectProxyConcurrencyParam] = envoy.DefaultConnectProxyConcurrency
	}
	if _, ok := node.Meta[envoy.DefaultTransparentProxyUIDParam]; !ok {
		node.Meta[envoy.DefaultTransparentProxyUIDParam] = envoy.DefaultTransparentProxyUID
	}
	if _, ok := node.Meta[envoy.DefaultTransparentProxyOutboundPortParam]; !ok {
		node.Meta[envoy.DefaultTransparentProxyOutboundPortParam] = envoy.DefaultTransparentProxyOutboundPort
	}
	// Set NodeMaxAllocs before dynamic configuration is set
	node.NodeMaxAllocs = newConfig.NodeMaxAllocs

	// Since node.Meta will get dynamic metadata merged in, save static metadata
	// here.
	c.metaStatic = maps.Clone(node.Meta)

	// Merge dynamic node metadata
	c.metaDynamic, err = c.stateDB.GetNodeMeta()
	if err != nil {
		return fmt.Errorf("error reading dynamic node metadata: %w", err)
	}

	if c.metaDynamic == nil {
		c.metaDynamic = map[string]*string{}
	}

	for dk, dv := range c.metaDynamic {
		if dv == nil {
			_, ok := node.Meta[dk]
			if ok {
				// Unset static node metadata
				delete(node.Meta, dk)
			} else {
				// Forget dynamic node metadata tombstone as there's no
				// static value to erase.
				delete(c.metaDynamic, dk)
			}
			continue

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Inspect the wrapped error to confirm it is a state DB/IO problem
2. Check disk space and permissions on data_dir
3. Stop the agent and run with a repaired/restored data_dir; if state.db is corrupt, back up and remove it (node re-registers with a new ID unless node.id is pinned)
4. Ensure only one Nomad client uses the data_dir

Example fix

// before
# nomad agent -client -data-dir=/shared/data  # shared with another agent
// after
# give each client its own data dir
# nomad agent -client -data-dir=/var/lib/nomad-client1

// preflight: ensure state db is openable before full setup
db, err := state.OpenStateDB(filepath.Join(cfg.DataDir, "state.db"))
if err != nil {
	return fmt.Errorf("state DB unusable: %w", err)
}
defer db.Close()

meta, err := stateDB.GetNodeMeta()
if err != nil {
	logger.Error("dynamic metadata unreadable", "err", err)
	return recoverStateDB() // backup state.db, reopen or recreate
}

When it happens

Trigger: c.stateDB.GetNodeMeta() returns an error during client initialization — typically a corrupt or locked bolt state DB, disk I/O failure, or incompatible state schema after a Nomad version upgrade.

Common situations: Disk full or failing; state.db corrupted after a crash/kill -9; upgrading Nomad with an old state DB; wrong permissions on the data_dir; running two agents against the same data_dir (bolt lock).

Related errors

• node ID setup failed: %v
• error syncing dynamic node metadata: %w
• error saving client identity: %w
• failed to remove alloc dir %q: %w
• Failed to make the alloc directory %v: %w

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/6b015499f7608d91. Report an issue: GitHub.