{"record":{"id":"8f1c3df51e944816","repo":"hashicorp/nomad","slug":"exited-hook-q-failed-v","errorCode":null,"errorMessage":"exited hook %q failed: %v","messagePattern":"exited hook %q failed: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"client/allocrunner/taskrunner/task_runner_hooks.go","lineNumber":435,"sourceCode":"\tvar merr multierror.Error\n\tfor _, hook := range tr.runnerHooks {\n\t\tpost, ok := hook.(interfaces.TaskExitedHook)\n\t\tif !ok {\n\t\t\tcontinue\n\t\t}\n\n\t\tname := post.Name()\n\t\tvar start time.Time\n\t\tif tr.logger.IsTrace() {\n\t\t\tstart = time.Now()\n\t\t\ttr.logger.Trace(\"running exited hook\", \"name\", name, \"start\", start)\n\t\t}\n\n\t\treq := interfaces.TaskExitedRequest{}\n\t\tvar resp interfaces.TaskExitedResponse\n\t\tif err := post.Exited(tr.killCtx, &req, &resp); err != nil {\n\t\t\ttr.emitHookError(err, name)\n\t\t\tmerr.Errors = append(merr.Errors, fmt.Errorf(\"exited hook %q failed: %v\", name, err))\n\t\t}\n\n\t\t// No need to persist as TaskExitedResponse is currently empty\n\n\t\tif tr.logger.IsTrace() {\n\t\t\tend := time.Now()\n\t\t\ttr.logger.Trace(\"finished exited hooks\", \"name\", name, \"end\", end, \"duration\", end.Sub(start))\n\t\t}\n\t}\n\n\treturn merr.ErrorOrNil()\n\n}\n\n// stop is used to run the stop hooks.\nfunc (tr *TaskRunner) stop() error {\n\tif tr.logger.IsTrace() {\n\t\tstart := time.Now()","sourceCodeStart":417,"sourceCodeEnd":453,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/client/allocrunner/taskrunner/task_runner_hooks.go#L417-L453","documentation":"This error wraps a failure from a task's exited hook, invoked after the task has finished running. During taskrunner Run teardown, each registered hook's Exited method is called with an empty TaskExitedRequest; a non-nil error is logged via emitHookError and accumulated in a multierror with this message. Since TaskExitedResponse is empty and not persisted, the failure affects cleanup/telemetry rather than task state.","triggerScenarios":"A hook implementing interfaces.TaskExited's Exited(ctx, req, resp) returns an error when the taskrunner processes task exit during Run — e.g. a driver's exit handling fails to release resources or a monitoring hook fails to report task exit.","commonSituations":"Driver plugins failing to clean up after task termination (leaked processes, failed stats close); hooks contacting external systems (Consul deregistration issues) after task exit; plugin crashes or timeouts triggered by task exit events; failures occurring alongside an already-failing task exit.","solutions":["Inspect the wrapped inner error to find which exited hook failed and why","Check driver/plugin logs for resource cleanup failures after the task exited","Verify external systems the hook talks to (Consul, driver daemons) are healthy","Ensure leftover task resources are cleaned (nomad system gc or restart the client) if cleanup failed","Upgrade or restart the driver plugin if it is crashing during exit handling"],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"// Go: wrap Run and treat exited-hook failures as cleanup warnings\nerr := tr.Run()\nif err != nil && strings.Contains(err.Error(), \"exited hook\") {\n    logger.Warn(\"exited hook failure during teardown; task state unaffected\", \"err\", err)\n    // trigger resource GC or plugin restart as remediation\n}","preventionTips":["Ensure driver plugins handle task exit idempotently and tolerate already-stopped workloads","Keep external systems used by exit hooks (Consul) healthy and ACL-authorized","Schedule periodic client GC to clean resources left by failed exit hooks","Watch plugin logs for crashes correlated with task exit events"],"tags":["nomad","task-runner","hooks","exited"],"backgroundTag":"task-lifecycle-hook-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T15:18:49.778Z"}