{"record":{"id":"f6b7476b50c0124b","repo":"hashicorp/nomad","slug":"failed-to-restart-task-s-v","errorCode":null,"errorMessage":"failed to restart task %s: %v","messagePattern":"failed to restart task (.+?): (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"client/allocrunner/alloc_runner.go","lineNumber":1459,"sourceCode":"\t\tdefer close(waitCh)\n\t\tfor tn, tr := range ar.tasks {\n\t\t\twg.Add(1)\n\t\t\tgo func(taskName string, taskRunner *taskrunner.TaskRunner) {\n\t\t\t\tdefer wg.Done()\n\n\t\t\t\tvar e error\n\t\t\t\tif force {\n\t\t\t\t\te = taskRunner.ForceRestart(ctx, event.Copy(), failure)\n\t\t\t\t} else {\n\t\t\t\t\te = taskRunner.Restart(ctx, event.Copy(), failure)\n\t\t\t\t}\n\n\t\t\t\t// Ignore ErrTaskNotRunning errors since tasks that are not\n\t\t\t\t// running are expected to not be restarted.\n\t\t\t\tif e != nil && e != te.ErrTaskNotRunning {\n\t\t\t\t\terrMutex.Lock()\n\t\t\t\t\tdefer errMutex.Unlock()\n\t\t\t\t\terr = multierror.Append(err, fmt.Errorf(\"failed to restart task %s: %v\", taskName, e))\n\t\t\t\t}\n\t\t\t}(tn, tr)\n\t\t}\n\t\twg.Wait()\n\t}()\n\n\tselect {\n\tcase <-waitCh:\n\tcase <-ctx.Done():\n\t}\n\n\treturn err.ErrorOrNil()\n}\n\n// Signal sends a signal request to task runners inside an allocation. If the\n// taskName is empty, then it is sent to all tasks.\nfunc (ar *allocRunner) Signal(taskName, signal string) error {\n\tevent := structs.NewTaskEvent(structs.TaskSignaling).SetSignalText(signal)","sourceCodeStart":1441,"sourceCodeEnd":1477,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/client/allocrunner/alloc_runner.go#L1441-L1477","documentation":"restartTasks restarts every task runner concurrently and aggregates per-task failures (ignoring ErrTaskNotRunning, since not-running tasks are legitimately skipped). This error wraps a per-task restart failure from tr.Restart for the named task, collected into the returned multierror.","triggerScenarios":"A task's Restart (context, event, failure=false) returns a non-nil error other than ErrTaskNotRunning during alloc-wide restart — e.g. driver restart failure, task runner shutting down concurrently, or state errors inside the task runner.","commonSituations":"Driver unable to restart the container/process (image removed, runtime daemon unhealthy like dockerd down); task in a state that rejects restart; alloc being concurrently stopped while a restart is issued.","solutions":["Inspect the wrapped %v per-task cause — it identifies the driver-level restart failure","Verify the task's driver runtime is healthy (e.g. `docker ps` works, driver plugin is running) and retry the restart","Use `nomad alloc restart <alloc>` again once transient driver issues clear, or reschedule the task via a new deployment","If the alloc is mid-teardown, don't restart — wait for terminal state and rely on rescheduling"],"exampleFix":null,"handlingStrategy":"retry","validationCode":"// Ensure the driver runtime is healthy before mass restarts\n// e.g. for docker:\nif err := exec.Command(\"docker\", \"info\").Run(); err != nil {\n  return fmt.Errorf(\"docker unhealthy, skipping restart: %w\", err)\n}\n// and confirm alloc is running:\nif alloc.ClientStatus != \"running\" { return fmt.Errorf(\"alloc not running\") }","typeGuard":null,"tryCatchPattern":"err := client.Allocations().Restart(allocID, \"\", nil)\nif err != nil {\n  if strings.Contains(err.Error(), \"failed to restart task\") {\n    // inspect wrapped cause; wait/backoff, then retry or reschedule\n    time.Sleep(10 * time.Second)\n  }\n}","preventionTips":["Check driver daemon health (dockerd, etc.) before bulk restarts","Ignore ErrTaskNotRunning outcomes — they are expected for stopped tasks","Avoid restarting allocs that are concurrently being stopped or rescheduled","Back off and retry restarts; if persistent, replace the alloc via redeployment"],"tags":["restart","task-runner","driver","concurrency"],"backgroundTag":"task-restart-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}