{"record":{"id":"6f948ca51bf53b2b","repo":"hashicorp/nomad","slug":"stop-hook-q-failed-v","errorCode":null,"errorMessage":"stop hook %q failed: %v","messagePattern":"stop hook %q failed: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"client/allocrunner/taskrunner/task_runner_hooks.go","lineNumber":488,"sourceCode":"\t\tif tr.logger.IsTrace() {\n\t\t\tstart = time.Now()\n\t\t\ttr.logger.Trace(\"running stop hook\", \"name\", name, \"start\", start)\n\t\t}\n\n\t\treq := interfaces.TaskStopRequest{\n\t\t\tTaskDir: tr.taskDir,\n\t\t}\n\n\t\torigHookState := tr.hookState(name)\n\t\tif origHookState != nil {\n\t\t\t// Give the hook data provided by prestart\n\t\t\treq.ExistingState = origHookState.Data\n\t\t}\n\n\t\tvar resp interfaces.TaskStopResponse\n\t\tif err := post.Stop(tr.killCtx, &req, &resp); err != nil {\n\t\t\ttr.emitHookError(err, name)\n\t\t\tmerr.Errors = append(merr.Errors, fmt.Errorf(\"stop hook %q failed: %v\", name, err))\n\t\t}\n\n\t\t// Stop hooks cannot alter state and must be idempotent, so\n\t\t// unlike prestart there's no state to persist here.\n\n\t\tif tr.logger.IsTrace() {\n\t\t\tend := time.Now()\n\t\t\ttr.logger.Trace(\"finished stop hook\", \"name\", name, \"end\", end, \"duration\", end.Sub(start))\n\t\t}\n\t}\n\n\treturn merr.ErrorOrNil()\n}\n\n// update is used to run the runners update hooks. Should only be called from\n// Run(). To trigger an update, update state on the TaskRunner and call\n// triggerUpdateHooks.\nfunc (tr *TaskRunner) updateHooks() {","sourceCodeStart":470,"sourceCodeEnd":506,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/client/allocrunner/taskrunner/task_runner_hooks.go#L470-L506","documentation":"This error wraps a failure from a task's stop hook, run during task shutdown/teardown. Each registered hook's Stop method is called with a TaskStopRequest (including prior hook state); errors are logged via emitHookError and added to a multierror with this message. Stop hooks must be idempotent and cannot alter state, so this failure does not block teardown but signals incomplete cleanup.","triggerScenarios":"A hook implementing Stop(ctx, req, resp) returns an error during taskrunner stop processing in Run — e.g. script check teardown fails, service deregistration fails, or a driver's stop path returns an error while killing the task.","commonSituations":"Consul/Vault deregistration failing during task stop due to network issues or ACL permission problems; driver plugins failing to kill/stop the underlying workload (container already gone, docker daemon errors); stop hooks racing with killCtx timeout when the task is being force-killed; plugin crashes during shutdown.","solutions":["Read the wrapped inner error to identify the failing stop hook and root cause","Check connectivity/ACLs for systems the hook touches at stop time (Consul deregistration, Vault token revoke)","Verify the driver can stop workloads (e.g. docker daemon healthy, container not already removed)","Manually clean leaked resources (orphaned containers, stale services) after the failed stop","If killCtx timeouts are the cause, review task kill_timeout and client shutdown sequencing"],"exampleFix":"// before: job with Consul service but client lacking Consul ACL write permission\nservice {\n  name = \"web\"\n}\n\n// after: grant the Nomad client's Consul token service:write ACL, or fix provider config\nservice {\n  name     = \"web\"\n  provider = \"consul\" // with proper ACL token configured on the client\n}","handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"// Go: intercept stop-hook failures from Run for cleanup handling\nif err := tr.Run(); err != nil {\n    if strings.Contains(err.Error(), \"stop hook\") {\n        logger.Warn(\"stop hook failed; manual cleanup may be required\", \"err\", err)\n        // deregister stale services / kill orphaned containers\n    }\n}","preventionTips":["Grant client tokens sufficient Consul/Vault ACLs for deregistration and revocation at stop time","Set a realistic kill_timeout so stop hooks are not cut off by killCtx cancellation","Keep stop hooks idempotent and tolerant of resources already removed","Periodically scan for orphaned containers/stale services to catch silent stop-hook failures"],"tags":["nomad","task-runner","hooks","stop"],"backgroundTag":"task-lifecycle-hook-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}