{"record":{"id":"2d16107ffd70d673","repo":"hashicorp/nomad","slug":"failed-to-read-plugin-state-v","errorCode":null,"errorMessage":"failed to read plugin state: %v","messagePattern":"failed to read plugin state: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"client/devicemanager/manager.go","lineNumber":276,"sourceCode":"\t\tif !i.HasDevices(d) {\n\t\t\tcontinue\n\t\t}\n\n\t\t// We found a match so reserve\n\t\treturn i.DeviceStats(d), nil\n\t}\n\n\treturn nil, UnknownDeviceErrFromAllocated(\"failed to collect statistics\", d)\n}\n\n// cleanupStalePlugins reads the device managers state and shuts down any\n// previously launched plugin.\nfunc (m *manager) cleanupStalePlugins() error {\n\n\t// Read the old plugin state\n\ts, err := m.state.GetDevicePluginState()\n\tif err != nil {\n\t\treturn fmt.Errorf(\"failed to read plugin state: %v\", err)\n\t}\n\n\t// No state was stored so there is nothing to do.\n\tif s == nil {\n\t\treturn nil\n\t}\n\n\t// For each plugin go through and try to shut it down\n\tvar mErr multierror.Error\n\tfor name, c := range s.ReattachConfigs {\n\t\trc, err := pstructs.ReattachConfigToGoPlugin(c)\n\t\tif err != nil {\n\t\t\t_ = multierror.Append(&mErr, fmt.Errorf(\"failed to convert reattach config: %v\", err))\n\t\t\tcontinue\n\t\t}\n\n\t\tinstance, err := m.loader.Reattach(name, base.PluginTypeDevice, rc)\n\t\tif err != nil {","sourceCodeStart":258,"sourceCodeEnd":294,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/client/devicemanager/manager.go#L258-L294","documentation":"cleanupStalePlugins (run at manager startup from Run) reads the persisted device plugin state via state.GetDevicePluginState. If reading state fails, the error is wrapped. The manager cannot determine which plugins were previously launched, so startup cleanup is aborted.","triggerScenarios":"Nomad client startup with a corrupted/unreadable bolt state DB or an I/O error while the state store reads the device plugin reattach-config bucket.","commonSituations":"Disk full or permissions problem on the client data_dir; corrupted state.db after crash; state directory restored from another node.","solutions":["Inspect the underlying error in client logs and fix disk/permission issues on data_dir.","Restart the nomad client after repairing the state store; bolt may recover a cleanly closed DB.","As a last resort (data loss acceptable), back up and remove the corrupted client state DB so a fresh one is created, then restart.","Ensure data_dir is on healthy writable storage before restarting."],"exampleFix":"// before: state db corrupt, agent crash-loops\n# check disk and perms\nsudo chown -R nomad:nomad /var/lib/nomad && ls -ld /var/lib/nomad/data\n// after: repair or reset state\nmv /var/lib/nomad/data /var/lib/nomad/data.bak && systemctl restart nomad","handlingStrategy":"fallback","validationCode":"// health check state dir at startup\nif fi, err := os.Stat(dataDir); err != nil || !fi.IsDir() { fixPermsOrCreate(dataDir) }\n","typeGuard":null,"tryCatchPattern":"if err := manager.Run(); err != nil && strings.Contains(err.Error(), \"failed to read plugin state\") {\n  // alert, repair data_dir/state.db, then restart client\n  log.Errorf(\"device state unreadable: %v\", err)\n}","preventionTips":["Monitor disk health and free space on the client data_dir volume","Ensure the nomad user owns data_dir and state.db","Shut down clients gracefully so bolt state is closed cleanly"],"tags":["devicemanager","state-store","startup"],"backgroundTag":"state-read-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}