hashicorp/nomad · error

failed to start task after driver exited unexpectedly: %v

Error message

failed to start task after driver exited unexpectedly: %v

What it means

In runDriver, after recovering from ErrPluginShutdown and successfully reinitializing the driver, the second StartTask attempt also failed; the task is marked TaskDriverFailure with this error and the run aborts.

Source

Thrown at client/allocrunner/taskrunner/task_runner.go:983

		}
		return nil
	}

	// Start the job if there's no existing handle (or if RecoverTask failed)
	handle, net, err := tr.driver.StartTask(taskConfig)
	if err != nil {
		// The plugin has died, try relaunching it
		if err == bstructs.ErrPluginShutdown {
			tr.logger.Info("failed to start task because plugin shutdown unexpectedly; attempting to recover")
			if err := tr.initDriver(); err != nil {
				taskErr := fmt.Errorf("failed to initialize driver after it exited unexpectedly: %v", err)
				tr.EmitEvent(structs.NewTaskEvent(structs.TaskDriverFailure).SetDriverError(taskErr))
				return taskErr
			}

			handle, net, err = tr.driver.StartTask(taskConfig)
			if err != nil {
				taskErr := fmt.Errorf("failed to start task after driver exited unexpectedly: %v", err)
				tr.EmitEvent(structs.NewTaskEvent(structs.TaskDriverFailure).SetDriverError(taskErr))
				return taskErr
			}
		} else {
			// Do *NOT* wrap the error here without maintaining whether or not is Recoverable.
			// You must emit a task event failure to be considered Recoverable
			tr.EmitEvent(structs.NewTaskEvent(structs.TaskDriverFailure).SetDriverError(err))
			return err
		}
	}

	tr.stateLock.Lock()
	tr.localState.TaskHandle = handle
	tr.localState.DriverNetwork = net
	if err := tr.stateDB.PutTaskRunnerLocalState(tr.allocID, tr.taskName, tr.localState); err != nil {
		//TODO Nomad will be unable to restore this task; try to kill
		//     it now and fail? In general we prefer to leave running
		//     tasks running even if the agent encounters an error.

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Read the wrapped second StartTask error for the root cause
2. Check driver health and plugin logs after the reinit
3. Verify task config and host resources (ports, volumes) are valid
4. Reschedule the task on a healthy node/client

When it happens

Trigger: Thrown at client/allocrunner/taskrunner/task_runner.go:983 when the library encounters an invalid state.

Common situations: See trigger scenarios.

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/97906d26bdce6cff. Report an issue: GitHub.