{"record":{"id":"6b015499f7608d91","repo":"hashicorp/nomad","slug":"error-reading-dynamic-node-metadata-w","errorCode":null,"errorMessage":"error reading dynamic node metadata: %w","messagePattern":"error reading dynamic node metadata: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"client/client.go","lineNumber":1715,"sourceCode":"\t\tnode.Meta[envoy.DefaultConnectProxyConcurrencyParam] = envoy.DefaultConnectProxyConcurrency\n\t}\n\tif _, ok := node.Meta[envoy.DefaultTransparentProxyUIDParam]; !ok {\n\t\tnode.Meta[envoy.DefaultTransparentProxyUIDParam] = envoy.DefaultTransparentProxyUID\n\t}\n\tif _, ok := node.Meta[envoy.DefaultTransparentProxyOutboundPortParam]; !ok {\n\t\tnode.Meta[envoy.DefaultTransparentProxyOutboundPortParam] = envoy.DefaultTransparentProxyOutboundPort\n\t}\n\t// Set NodeMaxAllocs before dynamic configuration is set\n\tnode.NodeMaxAllocs = newConfig.NodeMaxAllocs\n\n\t// Since node.Meta will get dynamic metadata merged in, save static metadata\n\t// here.\n\tc.metaStatic = maps.Clone(node.Meta)\n\n\t// Merge dynamic node metadata\n\tc.metaDynamic, err = c.stateDB.GetNodeMeta()\n\tif err != nil {\n\t\treturn fmt.Errorf(\"error reading dynamic node metadata: %w\", err)\n\t}\n\n\tif c.metaDynamic == nil {\n\t\tc.metaDynamic = map[string]*string{}\n\t}\n\n\tfor dk, dv := range c.metaDynamic {\n\t\tif dv == nil {\n\t\t\t_, ok := node.Meta[dk]\n\t\t\tif ok {\n\t\t\t\t// Unset static node metadata\n\t\t\t\tdelete(node.Meta, dk)\n\t\t\t} else {\n\t\t\t\t// Forget dynamic node metadata tombstone as there's no\n\t\t\t\t// static value to erase.\n\t\t\t\tdelete(c.metaDynamic, dk)\n\t\t\t}\n\t\t\tcontinue","sourceCodeStart":1697,"sourceCodeEnd":1733,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/client/client.go#L1697-L1733","documentation":"The client merges dynamic node metadata (tombstone-aware key/value metadata persisted in the local state DB) into the node during setup. If stateDB.GetNodeMeta fails to read that data, setup aborts with this wrapped error. It indicates local persistent state is unreadable, not a server-side problem.","triggerScenarios":"c.stateDB.GetNodeMeta() returns an error during client initialization — typically a corrupt or locked bolt state DB, disk I/O failure, or incompatible state schema after a Nomad version upgrade.","commonSituations":"Disk full or failing; state.db corrupted after a crash/kill -9; upgrading Nomad with an old state DB; wrong permissions on the data_dir; running two agents against the same data_dir (bolt lock).","solutions":["Inspect the wrapped error to confirm it is a state DB/IO problem","Check disk space and permissions on data_dir","Stop the agent and run with a repaired/restored data_dir; if state.db is corrupt, back up and remove it (node re-registers with a new ID unless node.id is pinned)","Ensure only one Nomad client uses the data_dir"],"exampleFix":"// before\n# nomad agent -client -data-dir=/shared/data  # shared with another agent\n// after\n# give each client its own data dir\n# nomad agent -client -data-dir=/var/lib/nomad-client1","handlingStrategy":"validation","validationCode":"// preflight: ensure state db is openable before full setup\ndb, err := state.OpenStateDB(filepath.Join(cfg.DataDir, \"state.db\"))\nif err != nil {\n\treturn fmt.Errorf(\"state DB unusable: %w\", err)\n}\ndefer db.Close()","typeGuard":null,"tryCatchPattern":"meta, err := stateDB.GetNodeMeta()\nif err != nil {\n\tlogger.Error(\"dynamic metadata unreadable\", \"err\", err)\n\treturn recoverStateDB() // backup state.db, reopen or recreate\n}","preventionTips":["Monitor disk space on data_dir volumes","Gracefully shut down Nomad (avoid kill -9 on state.db)","Run one agent per data_dir to avoid bolt lock contention","Test version upgrades on a staged client with a copy of state.db"],"tags":["nomad-client","state-db","metadata"],"backgroundTag":"state-db-read-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}