{"record":{"id":"59a3864831c665b2","repo":"hashicorp/nomad","slug":"error-syncing-dynamic-node-metadata-w","errorCode":null,"errorMessage":"error syncing dynamic node metadata: %w","messagePattern":"error syncing dynamic node metadata: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"client/client.go","lineNumber":1742,"sourceCode":"\t\t\t_, ok := node.Meta[dk]\n\t\t\tif ok {\n\t\t\t\t// Unset static node metadata\n\t\t\t\tdelete(node.Meta, dk)\n\t\t\t} else {\n\t\t\t\t// Forget dynamic node metadata tombstone as there's no\n\t\t\t\t// static value to erase.\n\t\t\t\tdelete(c.metaDynamic, dk)\n\t\t\t}\n\t\t\tcontinue\n\t\t}\n\n\t\tnode.Meta[dk] = *dv\n\t}\n\n\t// Write back dynamic node metadata as tombstones may have been removed\n\t// above\n\tif err := c.stateDB.PutNodeMeta(c.metaDynamic); err != nil {\n\t\treturn fmt.Errorf(\"error syncing dynamic node metadata: %w\", err)\n\t}\n\n\tif c.config.DefaultIneligible {\n\t\tnode.SchedulingEligibility = structs.NodeSchedulingIneligible\n\t} else {\n\t\tnode.SchedulingEligibility = structs.NodeSchedulingEligible\n\t}\n\n\tc.config = newConfig\n\treturn nil\n}\n\n// updateNodeFromFingerprint updates the node with the result of\n// fingerprinting the node from the diff that was created\nfunc (c *Client) updateNodeFromFingerprint(response *fingerprint.FingerprintResponse) *structs.Node {\n\tc.configLock.Lock()\n\tdefer c.configLock.Unlock()\n","sourceCodeStart":1724,"sourceCodeEnd":1760,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/client/client.go#L1724-L1760","documentation":"After reconciling tombstones in dynamic node metadata, the client writes the cleaned map back via stateDB.PutNodeMeta. Failure to persist this write aborts node setup with this wrapped error. It signals the client cannot update its local persistent store.","triggerScenarios":"c.stateDB.PutNodeMeta(c.metaDynamic) returns an error during client setup — corrupt/locked bolt DB, disk-full or read-only filesystem, or I/O error on the data_dir.","commonSituations":"Disk full or mounted read-only; data_dir permissions changed; state.db corrupted; filesystem errors after a crash.","solutions":["Read the wrapped error to confirm the storage-level cause","Verify data_dir has free space and is writable by the Nomad user (df -h, mount flags)","Repair the filesystem or restore the state DB; as a last resort stop Nomad and remove state.db (accepting node re-registration)","Check dmesg/journal for disk I/O errors and replace failing hardware"],"exampleFix":"# before\n/dev/sda1 on /var/lib/nomad type ext3 (ro)\n// after\nsudo mount -o remount,rw /var/lib/nomad\nsudo systemctl restart nomad","handlingStrategy":"validation","validationCode":"info, err := os.Stat(cfg.DataDir)\nif err != nil || !info.IsDir() {\n\treturn fmt.Errorf(\"data_dir missing or not a directory\")\n}\nif err := probeWrite(cfg.DataDir); err != nil {\n\treturn fmt.Errorf(\"data_dir not writable (read-only fs or permissions): %w\", err)\n}","typeGuard":null,"tryCatchPattern":"if err := stateDB.PutNodeMeta(meta); err != nil {\n\tlogger.Error(\"cannot persist node metadata\", \"err\", err)\n\talertOnDiskFull(err) // check ENOSPC / EROFS and page\n\treturn err\n}","preventionTips":["Alert on disk usage >85% on data_dir filesystems","Keep data_dir off read-only or ephemeral mounts","Ensure the Nomad user owns data_dir after image/config changes","Watch dmesg for I/O errors on the host"],"tags":["nomad-client","state-db","disk"],"backgroundTag":"state-db-write-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}