{"record":{"id":"5166106226979244","repo":"hashicorp/nomad","slug":"failed-to-reattach-to-plugin-q-v","errorCode":null,"errorMessage":"failed to reattach to plugin %q: %v","messagePattern":"failed to reattach to plugin %q: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"client/devicemanager/manager.go","lineNumber":295,"sourceCode":"\t}\n\n\t// No state was stored so there is nothing to do.\n\tif s == nil {\n\t\treturn nil\n\t}\n\n\t// For each plugin go through and try to shut it down\n\tvar mErr multierror.Error\n\tfor name, c := range s.ReattachConfigs {\n\t\trc, err := pstructs.ReattachConfigToGoPlugin(c)\n\t\tif err != nil {\n\t\t\t_ = multierror.Append(&mErr, fmt.Errorf(\"failed to convert reattach config: %v\", err))\n\t\t\tcontinue\n\t\t}\n\n\t\tinstance, err := m.loader.Reattach(name, base.PluginTypeDevice, rc)\n\t\tif err != nil {\n\t\t\t_ = multierror.Append(&mErr, fmt.Errorf(\"failed to reattach to plugin %q: %v\", name, err))\n\t\t\tcontinue\n\t\t}\n\n\t\t// Kill the instance\n\t\tinstance.Kill()\n\t}\n\n\treturn mErr.ErrorOrNil()\n}\n\n// storePluginReattachConfig is used as a callback to the instance managers and\n// persists thhe plugin reattach configurations.\nfunc (m *manager) storePluginReattachConfig(id loader.PluginID, c *plugin.ReattachConfig) error {\n\tm.reattachConfigLock.Lock()\n\tdefer m.reattachConfigLock.Unlock()\n\n\t// Store the new reattach config\n\tm.reattachConfigs[id] = pstructs.ReattachConfigFromGoPlugin(c)","sourceCodeStart":277,"sourceCodeEnd":313,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/client/devicemanager/manager.go#L277-L313","documentation":"cleanupStalePlugins reattaches to each previously-running device plugin via loader.Reattach so it can kill the stale instance. If reattaching fails (process gone, handshake fails, wrong plugin type), the error is appended to the multierror and the loop continues. The stale process could not be contacted or re-established as a plugin client.","triggerScenarios":"The old plugin process exited or its socket disappeared between client restart and cleanup; the stored PID no longer matches a live plugin; plugin binary changed protocol version.","commonSituations":"Node reboot or OOM-kill of plugin process before Nomad restarted; stale unix socket in data_dir; go-plugin handshake mismatch after upgrade.","solutions":["Confirm the stale plugin process is gone (ps) — if it is, this is benign and the plugin will be relaunched.","Remove stale plugin unix sockets from data_dir/plugin_*/ and restart the client.","Fix plugin binary/API version mismatch if the process exists but handshake fails.","Restart the nomad agent so cleanup completes and plugins relaunch."],"exampleFix":"// before: stale socket causes reattach failure\nls /var/lib/nomad/data/plugin_*/  # leftover sockets from killed plugin\n// after: clean stale sockets and restart\nrm /var/lib/nomad/data/plugin_*/* && systemctl restart nomad","handlingStrategy":"retry","validationCode":"// check whether the old plugin process still exists before expecting reattach\nif err := syscall.Kill(int(rc.Pid), 0); err != nil { /* process gone: cleanup is a no-op, relaunch */ }","typeGuard":null,"tryCatchPattern":"if err := m.cleanupStalePlugins(); err != nil {\n  log.Warnf(\"reattach to stale plugin failed (will relaunch): %v\", err)\n}","preventionTips":["Restart plugins and client together after upgrades to avoid version-skew handshakes","Clean leftover plugin sockets in data_dir on unclean shutdowns","Monitor for OOM kills of plugin processes"],"tags":["devicemanager","reattach","plugin"],"backgroundTag":"plugin-reattach-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}