hashicorp/nomad · error

failed to start plugin: %v

Error message

failed to start plugin: %v

What it means

The device manager instance dispenses a go-plugin device plugin. When the cached singleton has exited, it re-dispenses via the plugin loader; if that Dispense call fails, this error wraps it. The plugin binary could not be launched or reattached, so no device plugin instance is available.

Source

Thrown at client/devicemanager/instance.go:290

	i.pluginLock.Lock()
	defer i.pluginLock.Unlock()

	// See if we already have a running instance
	if i.plugin != nil && !i.plugin.Exited() {
		return i.device, nil
	}

	// Get an instance of the plugin
	pluginInstance, err := i.loader.Dispense(i.id.Name, i.id.PluginType, i.pluginConfig, i.logger)
	if err != nil {
		// Retry as the error just indicates the singleton has exited
		if err == singleton.SingletonPluginExited {
			pluginInstance, err = i.loader.Dispense(i.id.Name, i.id.PluginType, i.pluginConfig, i.logger)
		}

		// If we still have an error there is a real problem
		if err != nil {
			return nil, fmt.Errorf("failed to start plugin: %v", err)
		}
	}

	// Convert to a fingerprint plugin
	device, ok := pluginInstance.Plugin().(device.DevicePlugin)
	if !ok {
		pluginInstance.Kill()
		return nil, fmt.Errorf("plugin loaded does not implement the driver interface")
	}

	// Store the plugin and device
	i.plugin = pluginInstance
	i.device = device

	// Store the reattach config
	if c, ok := pluginInstance.ReattachConfig(); ok {
		i.storeReattach(c)
	}

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Check the plugin binary path exists and is executable on the client node.
2. Review the underlying loader error in client logs (usually a go-plugin handshake/exit message).
3. Fix the device plugin stanza/plugin_config in the client configuration and restart the nomad agent.
4. Redeploy or repair the device plugin task so Dispense can launch it.

Example fix

// before: client config pointing at missing binary
plugin "nvidia-gpu" { config { ... } }
// after: ensure binary exists at configured path
ls /opt/cni/nvidia-device-plugin && nomad agent -config client.hcl

// before starting client, verify plugin binary
if _, err := os.Stat(pluginPath); err != nil {
  return fmt.Errorf("device plugin binary missing: %w", err)
}

dev, err := instance.dispense()
if err != nil && strings.Contains(err.Error(), "failed to start plugin") {
  // inspect loader error, fix binary/config, restart client
  log.Errorf("device plugin %s could not launch: %v", name, err)
}

When it happens

Trigger: The device plugin process exited and re-dispensing fails: plugin binary missing/not executable, plugin config invalid, go-plugin handshake fails, or loader cannot launch the process.

Common situations: Nomad client restarted while plugin binary path changed; device plugin task (e.g. NVIDIA device plugin) crashed repeatedly; bad plugin_config in client config; binary removed by upgrade.

Related errors

• plugin loaded does not implement the driver interface
• failed to reattach to plugin %q: %v
• failed to start plugin: %v
• plugin not found
• plugin not executable

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/6128e03fa8dfcfd6. Report an issue: GitHub.