hashicorp/nomad · error

failed to restart task %s: %v

Error message

failed to restart task %s: %v

What it means

restartTasks restarts every task runner concurrently and aggregates per-task failures (ignoring ErrTaskNotRunning, since not-running tasks are legitimately skipped). This error wraps a per-task restart failure from tr.Restart for the named task, collected into the returned multierror.

Source

Thrown at client/allocrunner/alloc_runner.go:1459

		defer close(waitCh)
		for tn, tr := range ar.tasks {
			wg.Add(1)
			go func(taskName string, taskRunner *taskrunner.TaskRunner) {
				defer wg.Done()

				var e error
				if force {
					e = taskRunner.ForceRestart(ctx, event.Copy(), failure)
				} else {
					e = taskRunner.Restart(ctx, event.Copy(), failure)
				}

				// Ignore ErrTaskNotRunning errors since tasks that are not
				// running are expected to not be restarted.
				if e != nil && e != te.ErrTaskNotRunning {
					errMutex.Lock()
					defer errMutex.Unlock()
					err = multierror.Append(err, fmt.Errorf("failed to restart task %s: %v", taskName, e))
				}
			}(tn, tr)
		}
		wg.Wait()
	}()

	select {
	case <-waitCh:
	case <-ctx.Done():
	}

	return err.ErrorOrNil()
}

// Signal sends a signal request to task runners inside an allocation. If the
// taskName is empty, then it is sent to all tasks.
func (ar *allocRunner) Signal(taskName, signal string) error {
	event := structs.NewTaskEvent(structs.TaskSignaling).SetSignalText(signal)

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Inspect the wrapped %v per-task cause — it identifies the driver-level restart failure
2. Verify the task's driver runtime is healthy (e.g. `docker ps` works, driver plugin is running) and retry the restart
3. Use `nomad alloc restart <alloc>` again once transient driver issues clear, or reschedule the task via a new deployment
4. If the alloc is mid-teardown, don't restart — wait for terminal state and rely on rescheduling

// Ensure the driver runtime is healthy before mass restarts
// e.g. for docker:
if err := exec.Command("docker", "info").Run(); err != nil {
  return fmt.Errorf("docker unhealthy, skipping restart: %w", err)
}
// and confirm alloc is running:
if alloc.ClientStatus != "running" { return fmt.Errorf("alloc not running") }

err := client.Allocations().Restart(allocID, "", nil)
if err != nil {
  if strings.Contains(err.Error(), "failed to restart task") {
    // inspect wrapped cause; wait/backoff, then retry or reschedule
    time.Sleep(10 * time.Second)
  }
}

When it happens

Trigger: A task's Restart (context, event, failure=false) returns a non-nil error other than ErrTaskNotRunning during alloc-wide restart — e.g. driver restart failure, task runner shutting down concurrently, or state errors inside the task runner.

Common situations: Driver unable to restart the container/process (image removed, runtime daemon unhealthy like dockerd down); task in a state that rejects restart; alloc being concurrently stopped while a restart is issued.

Related errors

• failed creating runner for task %q: %v
• Failed to restart task %q: %w
• failed to parse config:
• ErrVolumeNameExists
• ErrAllocBroadcasterClosed

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/f6b7476b50c0124b. Report an issue: GitHub.