{"record":{"id":"096253731d7e8c75","repo":"hashicorp/nomad","slug":"failed-to-start-plugin-v-096253","errorCode":null,"errorMessage":"failed to start plugin: %v","messagePattern":"failed to start plugin: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"client/pluginmanager/drivermanager/instance.go","lineNumber":219,"sourceCode":"\t\t// If reattachment fails, get a new plugin instance\n\t\tif err != nil {\n\t\t\ti.logger.Warn(\"failed to reattach to plugin, starting new instance\", \"error\", err)\n\t\t\tpluginInstance, err = dispenseFn()\n\t\t}\n\t} else {\n\t\t// Get an instance of the plugin\n\t\tpluginInstance, err = dispenseFn()\n\t}\n\n\tif err != nil {\n\t\t// Retry as the error just indicates the singleton has exited\n\t\tif err == singleton.SingletonPluginExited {\n\t\t\tpluginInstance, err = dispenseFn()\n\t\t}\n\n\t\t// If we still have an error there is a real problem\n\t\tif err != nil {\n\t\t\treturn nil, fmt.Errorf(\"failed to start plugin: %v\", err)\n\t\t}\n\t}\n\n\t// Convert to a driver plugin\n\tdriver, ok := pluginInstance.Plugin().(drivers.DriverPlugin)\n\tif !ok {\n\t\tpluginInstance.Kill()\n\t\treturn nil, fmt.Errorf(\"plugin loaded does not implement the driver interface\")\n\t}\n\n\t// Initialize the plugin if it supports it.\n\tif initer, ok := driver.(drivers.DriverIniter); ok {\n\t\tif err := initer.Init(i.ctx); err != nil {\n\t\t\tpluginInstance.Kill()\n\t\t\treturn nil, fmt.Errorf(\"init of plugin %s failed: %w\", i.id, err)\n\t\t}\n\t}\n","sourceCodeStart":201,"sourceCodeEnd":237,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/client/pluginmanager/drivermanager/instance.go#L201-L237","documentation":"In Nomad's driver manager, dispense launches the driver plugin via a go-plugin singleton launcher. If the first dispense fails (e.g. SingletonPluginExited) it retries once; if the retry still errors, there is a real problem and the error is wrapped as 'failed to start plugin: %v'.","triggerScenarios":"dispenseFn() returns an error on both attempts in instance.dispense — plugin binary fails to launch, crashes at startup, exits prematurely, or the plugin handshake times out.","commonSituations":"Driver plugin binary missing/segfaulting; incompatible plugin (wrong go-plugin version/protocol); driver container OOM-killed; GPU/exec environment misconfigured; plugin stdout/stderr blocked causing handshake failure.","solutions":["Read the wrapped %v error and the driver plugin's stderr logs (client debug logs) to find the launch failure cause","Verify the driver is built into/available to the Nomad client and that the plugin binary is executable and matches the Nomad plugin protocol version","Check node resources (memory, CPU, GPU) for OOM/resource exhaustion of the plugin process","Restart the Nomad client or the affected allocation; if persistent, upgrade the driver plugin/Nomad to compatible versions"],"exampleFix":"// before: docker driver task with missing image config\nconfig { image = \"\" }\n// after\nconfig { image = \"nginx:1.25\" }  # plugin launches task successfully; dispense error gone","handlingStrategy":"retry","validationCode":"// precheck: plugin binary exists and is executable\nfi, err := os.Stat(pluginPath)\nif err != nil || fi.Mode()&0111 == 0 {\n    return fmt.Errorf(\"plugin %s missing or not executable\", pluginPath)\n}","typeGuard":null,"tryCatchPattern":"drv, err := mgr.Dispense(driverName)\nif err != nil {\n    if strings.HasPrefix(err.Error(), \"failed to start plugin\") {\n        // read plugin stderr from client logs, fix env, then retry once\n        time.Sleep(2 * time.Second)\n        drv, err = mgr.Dispense(driverName)\n    }\n    if err != nil { return err }\n}","preventionTips":["Keep driver plugin binaries compatible with the Nomad client's go-plugin protocol version","Check node resources to avoid plugin OOM kills","Capture and review plugin stderr in client debug logs","Pin Nomad and driver plugin versions to tested combinations"],"tags":["nomad","plugin","driver","go-plugin","process-launch"],"backgroundTag":"plugin-start-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}