hashicorp/nomad · warning

stop hook %q failed: %v

Error message

stop hook %q failed: %v

What it means

This error wraps a failure from a task's stop hook, run during task shutdown/teardown. Each registered hook's Stop method is called with a TaskStopRequest (including prior hook state); errors are logged via emitHookError and added to a multierror with this message. Stop hooks must be idempotent and cannot alter state, so this failure does not block teardown but signals incomplete cleanup.

Source

Thrown at client/allocrunner/taskrunner/task_runner_hooks.go:488

		if tr.logger.IsTrace() {
			start = time.Now()
			tr.logger.Trace("running stop hook", "name", name, "start", start)
		}

		req := interfaces.TaskStopRequest{
			TaskDir: tr.taskDir,
		}

		origHookState := tr.hookState(name)
		if origHookState != nil {
			// Give the hook data provided by prestart
			req.ExistingState = origHookState.Data
		}

		var resp interfaces.TaskStopResponse
		if err := post.Stop(tr.killCtx, &req, &resp); err != nil {
			tr.emitHookError(err, name)
			merr.Errors = append(merr.Errors, fmt.Errorf("stop hook %q failed: %v", name, err))
		}

		// Stop hooks cannot alter state and must be idempotent, so
		// unlike prestart there's no state to persist here.

		if tr.logger.IsTrace() {
			end := time.Now()
			tr.logger.Trace("finished stop hook", "name", name, "end", end, "duration", end.Sub(start))
		}
	}

	return merr.ErrorOrNil()
}

// update is used to run the runners update hooks. Should only be called from
// Run(). To trigger an update, update state on the TaskRunner and call
// triggerUpdateHooks.
func (tr *TaskRunner) updateHooks() {

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Read the wrapped inner error to identify the failing stop hook and root cause
2. Check connectivity/ACLs for systems the hook touches at stop time (Consul deregistration, Vault token revoke)
3. Verify the driver can stop workloads (e.g. docker daemon healthy, container not already removed)
4. Manually clean leaked resources (orphaned containers, stale services) after the failed stop
5. If killCtx timeouts are the cause, review task kill_timeout and client shutdown sequencing

Example fix

// before: job with Consul service but client lacking Consul ACL write permission
service {
  name = "web"
}

// after: grant the Nomad client's Consul token service:write ACL, or fix provider config
service {
  name     = "web"
  provider = "consul" // with proper ACL token configured on the client
}

// Go: intercept stop-hook failures from Run for cleanup handling
if err := tr.Run(); err != nil {
    if strings.Contains(err.Error(), "stop hook") {
        logger.Warn("stop hook failed; manual cleanup may be required", "err", err)
        // deregister stale services / kill orphaned containers
    }
}

When it happens

Trigger: A hook implementing Stop(ctx, req, resp) returns an error during taskrunner stop processing in Run — e.g. script check teardown fails, service deregistration fails, or a driver's stop path returns an error while killing the task.

Common situations: Consul/Vault deregistration failing during task stop due to network issues or ACL permission problems; driver plugins failing to kill/stop the underlying workload (container already gone, docker daemon errors); stop hooks racing with killCtx timeout when the task is being force-killed; plugin crashes during shutdown.

Related errors

• poststart hook %q failed: %v
• exited hook %q failed: %v
• failed to parse config:
• pre-run hook %q failed: %v
• update hook %q failed: %v

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/6f948ca51bf53b2b. Report an issue: GitHub.