hashicorp/nomad · warning

failed to reattach to plugin %q: %v

Error message

failed to reattach to plugin %q: %v

What it means

cleanupStalePlugins reattaches to each previously-running device plugin via loader.Reattach so it can kill the stale instance. If reattaching fails (process gone, handshake fails, wrong plugin type), the error is appended to the multierror and the loop continues. The stale process could not be contacted or re-established as a plugin client.

Source

Thrown at client/devicemanager/manager.go:295

	}

	// No state was stored so there is nothing to do.
	if s == nil {
		return nil
	}

	// For each plugin go through and try to shut it down
	var mErr multierror.Error
	for name, c := range s.ReattachConfigs {
		rc, err := pstructs.ReattachConfigToGoPlugin(c)
		if err != nil {
			_ = multierror.Append(&mErr, fmt.Errorf("failed to convert reattach config: %v", err))
			continue
		}

		instance, err := m.loader.Reattach(name, base.PluginTypeDevice, rc)
		if err != nil {
			_ = multierror.Append(&mErr, fmt.Errorf("failed to reattach to plugin %q: %v", name, err))
			continue
		}

		// Kill the instance
		instance.Kill()
	}

	return mErr.ErrorOrNil()
}

// storePluginReattachConfig is used as a callback to the instance managers and
// persists thhe plugin reattach configurations.
func (m *manager) storePluginReattachConfig(id loader.PluginID, c *plugin.ReattachConfig) error {
	m.reattachConfigLock.Lock()
	defer m.reattachConfigLock.Unlock()

	// Store the new reattach config
	m.reattachConfigs[id] = pstructs.ReattachConfigFromGoPlugin(c)

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Confirm the stale plugin process is gone (ps) — if it is, this is benign and the plugin will be relaunched.
2. Remove stale plugin unix sockets from data_dir/plugin_*/ and restart the client.
3. Fix plugin binary/API version mismatch if the process exists but handshake fails.
4. Restart the nomad agent so cleanup completes and plugins relaunch.

Example fix

// before: stale socket causes reattach failure
ls /var/lib/nomad/data/plugin_*/  # leftover sockets from killed plugin
// after: clean stale sockets and restart
rm /var/lib/nomad/data/plugin_*/* && systemctl restart nomad

// check whether the old plugin process still exists before expecting reattach
if err := syscall.Kill(int(rc.Pid), 0); err != nil { /* process gone: cleanup is a no-op, relaunch */ }

if err := m.cleanupStalePlugins(); err != nil {
  log.Warnf("reattach to stale plugin failed (will relaunch): %v", err)
}

When it happens

Trigger: The old plugin process exited or its socket disappeared between client restart and cleanup; the stored PID no longer matches a live plugin; plugin binary changed protocol version.

Common situations: Node reboot or OOM-kill of plugin process before Nomad restarted; stale unix socket in data_dir; go-plugin handshake mismatch after upgrade.

Related errors

• failed to start plugin: %v
• plugin loaded does not implement the driver interface
• failed to convert reattach config: %v
• failed to build ReattachConfig from task state: %v
• failed to reattach to executor: %v

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/5166106226979244. Report an issue: GitHub.