hashicorp/nomad · error

error syncing dynamic node metadata: %w

Error message

error syncing dynamic node metadata: %w

What it means

After reconciling tombstones in dynamic node metadata, the client writes the cleaned map back via stateDB.PutNodeMeta. Failure to persist this write aborts node setup with this wrapped error. It signals the client cannot update its local persistent store.

Source

Thrown at client/client.go:1742

			_, ok := node.Meta[dk]
			if ok {
				// Unset static node metadata
				delete(node.Meta, dk)
			} else {
				// Forget dynamic node metadata tombstone as there's no
				// static value to erase.
				delete(c.metaDynamic, dk)
			}
			continue
		}

		node.Meta[dk] = *dv
	}

	// Write back dynamic node metadata as tombstones may have been removed
	// above
	if err := c.stateDB.PutNodeMeta(c.metaDynamic); err != nil {
		return fmt.Errorf("error syncing dynamic node metadata: %w", err)
	}

	if c.config.DefaultIneligible {
		node.SchedulingEligibility = structs.NodeSchedulingIneligible
	} else {
		node.SchedulingEligibility = structs.NodeSchedulingEligible
	}

	c.config = newConfig
	return nil
}

// updateNodeFromFingerprint updates the node with the result of
// fingerprinting the node from the diff that was created
func (c *Client) updateNodeFromFingerprint(response *fingerprint.FingerprintResponse) *structs.Node {
	c.configLock.Lock()
	defer c.configLock.Unlock()

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Read the wrapped error to confirm the storage-level cause
2. Verify data_dir has free space and is writable by the Nomad user (df -h, mount flags)
3. Repair the filesystem or restore the state DB; as a last resort stop Nomad and remove state.db (accepting node re-registration)
4. Check dmesg/journal for disk I/O errors and replace failing hardware

Example fix

# before
/dev/sda1 on /var/lib/nomad type ext3 (ro)
// after
sudo mount -o remount,rw /var/lib/nomad
sudo systemctl restart nomad

info, err := os.Stat(cfg.DataDir)
if err != nil || !info.IsDir() {
	return fmt.Errorf("data_dir missing or not a directory")
}
if err := probeWrite(cfg.DataDir); err != nil {
	return fmt.Errorf("data_dir not writable (read-only fs or permissions): %w", err)
}

if err := stateDB.PutNodeMeta(meta); err != nil {
	logger.Error("cannot persist node metadata", "err", err)
	alertOnDiskFull(err) // check ENOSPC / EROFS and page
	return err
}

When it happens

Trigger: c.stateDB.PutNodeMeta(c.metaDynamic) returns an error during client setup — corrupt/locked bolt DB, disk-full or read-only filesystem, or I/O error on the data_dir.

Common situations: Disk full or mounted read-only; data_dir permissions changed; state.db corrupted; filesystem errors after a crash.

Related errors

• node ID setup failed: %v
• error reading dynamic node metadata: %w
• error saving client identity: %w
• Task can't ask for disk resources, they have to be specified
• failed to remove alloc dir %q: %w

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/59a3864831c665b2. Report an issue: GitHub.