hashicorp/nomad · error

failed to start plugin: %v

Error message

failed to start plugin: %v

What it means

In Nomad's driver manager, dispense launches the driver plugin via a go-plugin singleton launcher. If the first dispense fails (e.g. SingletonPluginExited) it retries once; if the retry still errors, there is a real problem and the error is wrapped as 'failed to start plugin: %v'.

Source

Thrown at client/pluginmanager/drivermanager/instance.go:219

		// If reattachment fails, get a new plugin instance
		if err != nil {
			i.logger.Warn("failed to reattach to plugin, starting new instance", "error", err)
			pluginInstance, err = dispenseFn()
		}
	} else {
		// Get an instance of the plugin
		pluginInstance, err = dispenseFn()
	}

	if err != nil {
		// Retry as the error just indicates the singleton has exited
		if err == singleton.SingletonPluginExited {
			pluginInstance, err = dispenseFn()
		}

		// If we still have an error there is a real problem
		if err != nil {
			return nil, fmt.Errorf("failed to start plugin: %v", err)
		}
	}

	// Convert to a driver plugin
	driver, ok := pluginInstance.Plugin().(drivers.DriverPlugin)
	if !ok {
		pluginInstance.Kill()
		return nil, fmt.Errorf("plugin loaded does not implement the driver interface")
	}

	// Initialize the plugin if it supports it.
	if initer, ok := driver.(drivers.DriverIniter); ok {
		if err := initer.Init(i.ctx); err != nil {
			pluginInstance.Kill()
			return nil, fmt.Errorf("init of plugin %s failed: %w", i.id, err)
		}
	}

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Read the wrapped %v error and the driver plugin's stderr logs (client debug logs) to find the launch failure cause
2. Verify the driver is built into/available to the Nomad client and that the plugin binary is executable and matches the Nomad plugin protocol version
3. Check node resources (memory, CPU, GPU) for OOM/resource exhaustion of the plugin process
4. Restart the Nomad client or the affected allocation; if persistent, upgrade the driver plugin/Nomad to compatible versions

Example fix

// before: docker driver task with missing image config
config { image = "" }
// after
config { image = "nginx:1.25" }  # plugin launches task successfully; dispense error gone

// precheck: plugin binary exists and is executable
fi, err := os.Stat(pluginPath)
if err != nil || fi.Mode()&0111 == 0 {
    return fmt.Errorf("plugin %s missing or not executable", pluginPath)
}

drv, err := mgr.Dispense(driverName)
if err != nil {
    if strings.HasPrefix(err.Error(), "failed to start plugin") {
        // read plugin stderr from client logs, fix env, then retry once
        time.Sleep(2 * time.Second)
        drv, err = mgr.Dispense(driverName)
    }
    if err != nil { return err }
}

When it happens

Trigger: dispenseFn() returns an error on both attempts in instance.dispense — plugin binary fails to launch, crashes at startup, exits prematurely, or the plugin handshake times out.

Common situations: Driver plugin binary missing/segfaulting; incompatible plugin (wrong go-plugin version/protocol); driver container OOM-killed; GPU/exec environment misconfigured; plugin stdout/stderr blocked causing handshake failure.

Related errors

• plugin is shut down
• could not validate task driver capabilities: %v
• plugin loaded does not implement the driver interface
• init of plugin %s failed: %w
• ErrDriverNotFound

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/096253731d7e8c75. Report an issue: GitHub.