{"record":{"id":"6675a249f95862d5","repo":"hashicorp/nomad","slug":"failed-to-convert-reattach-config-v","errorCode":null,"errorMessage":"failed to convert reattach config: %v","messagePattern":"failed to convert reattach config: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"client/devicemanager/manager.go","lineNumber":289,"sourceCode":"func (m *manager) cleanupStalePlugins() error {\n\n\t// Read the old plugin state\n\ts, err := m.state.GetDevicePluginState()\n\tif err != nil {\n\t\treturn fmt.Errorf(\"failed to read plugin state: %v\", err)\n\t}\n\n\t// No state was stored so there is nothing to do.\n\tif s == nil {\n\t\treturn nil\n\t}\n\n\t// For each plugin go through and try to shut it down\n\tvar mErr multierror.Error\n\tfor name, c := range s.ReattachConfigs {\n\t\trc, err := pstructs.ReattachConfigToGoPlugin(c)\n\t\tif err != nil {\n\t\t\t_ = multierror.Append(&mErr, fmt.Errorf(\"failed to convert reattach config: %v\", err))\n\t\t\tcontinue\n\t\t}\n\n\t\tinstance, err := m.loader.Reattach(name, base.PluginTypeDevice, rc)\n\t\tif err != nil {\n\t\t\t_ = multierror.Append(&mErr, fmt.Errorf(\"failed to reattach to plugin %q: %v\", name, err))\n\t\t\tcontinue\n\t\t}\n\n\t\t// Kill the instance\n\t\tinstance.Kill()\n\t}\n\n\treturn mErr.ErrorOrNil()\n}\n\n// storePluginReattachConfig is used as a callback to the instance managers and\n// persists thhe plugin reattach configurations.","sourceCodeStart":271,"sourceCodeEnd":307,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/client/devicemanager/manager.go#L271-L307","documentation":"During cleanupStalePlugins, each stored reattach config is converted to a go-plugin ReattachConfig via pstructs.ReattachConfigToGoPlugin. If conversion fails (malformed stored config), the error is appended to a multierror and cleanup continues with the next plugin. It indicates stale or corrupted reattach metadata persisted by a previous client run.","triggerScenarios":"The persisted ReattachConfigs map contains an entry with invalid/empty fields (bad PID, protocol, or address) that cannot be converted to a go-plugin reattach config.","commonSituations":"Client crashed mid-write leaving partial plugin state; state file hand-edited or from a different Nomad version with an incompatible reattach config schema.","solutions":["Inspect the stored device plugin state; delete stale reattach entries so the plugin is relaunched fresh.","Restart the nomad client — after cleanup the manager relaunches plugins and rewrites valid state.","Fix data_dir permissions/corruption if the state is repeatedly malformed.","Upgrade to a Nomad version that prunes stale reattach configs on shutdown."],"exampleFix":"// before: stale reattach entry blocks startup cleanup\n// state.ReattachConfigs: {\"nvidia-gpu\": {Pid: -1, ...}}\n// after: clear stale plugin state and restart client\nnomad client reload  # or remove the device-plugin state entry and restart nomad","handlingStrategy":"fallback","validationCode":"// validate reattach config fields before use\nfunc validReattach(c pstructs.ReattachConfig) bool { return c.Pid > 0 && c.Addr != nil && c.Protocol != \"\" }","typeGuard":"func isConvertible(c interface{}) bool {\n  _, err := pstructs.ReattachConfigToGoPlugin(c)\n  return err == nil\n}","tryCatchPattern":"if err := m.cleanupStalePlugins(); err != nil {\n  // multierror may contain conversion/reattach issues; log and relaunch plugins fresh\n  log.Warnf(\"stale plugin cleanup issues (non-fatal): %v\", err)\n}","preventionTips":["Shut down the client gracefully so reattach configs are pruned","Avoid hand-editing or cross-version copying of client state","Keep plugin binaries stable across client restarts so reattach succeeds"],"tags":["devicemanager","reattach","state-store"],"backgroundTag":"reattach-config-invalid","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}