{"record":{"id":"6128e03fa8dfcfd6","repo":"hashicorp/nomad","slug":"failed-to-start-plugin-v","errorCode":null,"errorMessage":"failed to start plugin: %v","messagePattern":"failed to start plugin: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"client/devicemanager/instance.go","lineNumber":290,"sourceCode":"\ti.pluginLock.Lock()\n\tdefer i.pluginLock.Unlock()\n\n\t// See if we already have a running instance\n\tif i.plugin != nil && !i.plugin.Exited() {\n\t\treturn i.device, nil\n\t}\n\n\t// Get an instance of the plugin\n\tpluginInstance, err := i.loader.Dispense(i.id.Name, i.id.PluginType, i.pluginConfig, i.logger)\n\tif err != nil {\n\t\t// Retry as the error just indicates the singleton has exited\n\t\tif err == singleton.SingletonPluginExited {\n\t\t\tpluginInstance, err = i.loader.Dispense(i.id.Name, i.id.PluginType, i.pluginConfig, i.logger)\n\t\t}\n\n\t\t// If we still have an error there is a real problem\n\t\tif err != nil {\n\t\t\treturn nil, fmt.Errorf(\"failed to start plugin: %v\", err)\n\t\t}\n\t}\n\n\t// Convert to a fingerprint plugin\n\tdevice, ok := pluginInstance.Plugin().(device.DevicePlugin)\n\tif !ok {\n\t\tpluginInstance.Kill()\n\t\treturn nil, fmt.Errorf(\"plugin loaded does not implement the driver interface\")\n\t}\n\n\t// Store the plugin and device\n\ti.plugin = pluginInstance\n\ti.device = device\n\n\t// Store the reattach config\n\tif c, ok := pluginInstance.ReattachConfig(); ok {\n\t\ti.storeReattach(c)\n\t}","sourceCodeStart":272,"sourceCodeEnd":308,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/client/devicemanager/instance.go#L272-L308","documentation":"The device manager instance dispenses a go-plugin device plugin. When the cached singleton has exited, it re-dispenses via the plugin loader; if that Dispense call fails, this error wraps it. The plugin binary could not be launched or reattached, so no device plugin instance is available.","triggerScenarios":"The device plugin process exited and re-dispensing fails: plugin binary missing/not executable, plugin config invalid, go-plugin handshake fails, or loader cannot launch the process.","commonSituations":"Nomad client restarted while plugin binary path changed; device plugin task (e.g. NVIDIA device plugin) crashed repeatedly; bad plugin_config in client config; binary removed by upgrade.","solutions":["Check the plugin binary path exists and is executable on the client node.","Review the underlying loader error in client logs (usually a go-plugin handshake/exit message).","Fix the device plugin stanza/plugin_config in the client configuration and restart the nomad agent.","Redeploy or repair the device plugin task so Dispense can launch it."],"exampleFix":"// before: client config pointing at missing binary\nplugin \"nvidia-gpu\" { config { ... } }\n// after: ensure binary exists at configured path\nls /opt/cni/nvidia-device-plugin && nomad agent -config client.hcl","handlingStrategy":"try-catch","validationCode":"// before starting client, verify plugin binary\nif _, err := os.Stat(pluginPath); err != nil {\n  return fmt.Errorf(\"device plugin binary missing: %w\", err)\n}","typeGuard":null,"tryCatchPattern":"dev, err := instance.dispense()\nif err != nil && strings.Contains(err.Error(), \"failed to start plugin\") {\n  // inspect loader error, fix binary/config, restart client\n  log.Errorf(\"device plugin %s could not launch: %v\", name, err)\n}","preventionTips":["Pin and verify the plugin binary path in client config before deployment","Keep the device plugin task supervised so it relaunches after crashes","Match go-plugin protocol versions between plugin binary and Nomad client"],"tags":["plugin","devicemanager","go-plugin"],"backgroundTag":"plugin-dispense-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}