hashicorp/nomad · error

Failed to signal container %q while killing: %v

Error message

Failed to signal container %q while killing: %v

What it means

During Kill, after the initial signal the driver waits for the container to exit and, if needed, signals it again. If that signaling call fails with an error that is not errdefs.IsNotModified (i.e. the container is genuinely still running and rejecting the signal), Kill returns this wrapped error naming the container ID.

Source

Thrown at drivers/docker/handle.go:197

		}

		ctx, cancel := context.WithTimeout(context.Background(), killTimeout)
		defer cancel()

		if err := h.Signal(ctx, signal); err != nil {
			// Container has already been removed.
			if errdefs.IsNotFound(err) {
				h.logger.Debug("attempted to signal nonexistent container")
				return nil
			}
			// Container has already been stopped.
			if errdefs.IsNotModified(err) {
				h.logger.Debug("attempted to signal a not-running container")
				return nil
			}

			h.logger.Error("failed to signal container while killing", "error", err)
			return fmt.Errorf("Failed to signal container %q while killing: %v", h.containerID, err)
		}

		select {
		case <-h.waitCh:
			return nil
		case <-ctx.Done():
		}

		// Stop the container forcefully.
		_, err = h.dockerClient.ContainerStop(context.Background(), h.containerID, mclient.ContainerStopOptions{Timeout: new(0)})
	}

	if err != nil {
		// Container has already been removed.
		if errdefs.IsNotFound(err) {
			h.logger.Debug("attempted to stop nonexistent container")
			return nil
		}

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Check Docker daemon health and logs on the host; restart if it is unresponsive.
2. Verify the container still exists ('docker ps -a | grep <id>') — external removal causes no-such-container errors.
3. Check Nomad client's access to the Docker socket.
4. Retry the alloc stop; if the container already died, the error will disappear.

Example fix

// before: fighting an unhealthy daemon during kill
nomad alloc stop <alloc>  # fails: Failed to signal container ...
// after: heal the daemon first
sudo systemctl restart docker && nomad alloc stop <alloc>

err := client.Nodes.StopAlloc(ctx, allocID, nil)
if err != nil && strings.Contains(err.Error(), "Failed to signal container") {
    // container may have died anyway; verify before escalating
    st, serr := client.Allocations().Info(ctx, allocID, nil)
    if serr == nil && st.TaskStates[task].Finished { return nil }
    return err
}

When it happens

Trigger: h.dockerClient.ContainerKill (via h.Signal) during kill returns a real API error — daemon connection failure, no such container (removed externally), permission denied — and the error is not 'not modified'.

Common situations: Docker daemon restarted mid-kill; container removed by an external supervisor while Nomad was killing it; socket permission changes; engine unresponsive under load.

Related errors

• failed to create exec object: %v
• failed to parse signal: %v
• Failed to stop container %s: %s
• Task not found
• Failed to signal task: %s, err: %v

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/b7fd7636b118edcc. Report an issue: GitHub.