hashicorp/nomad · error

Failed to stop container %s: %s

Error message

Failed to stop container %s: %s

What it means

When killing a task, the driver eventually issues a ContainerStop to shut the container down. If stop fails with any error other than 'not modified' (already stopped), the driver logs it and returns 'Failed to stop container <id>: <err>'. The task was not confirmed stopped.

Source

Thrown at drivers/docker/handle.go:223

		// Stop the container forcefully.
		_, err = h.dockerClient.ContainerStop(context.Background(), h.containerID, mclient.ContainerStopOptions{Timeout: new(0)})
	}

	if err != nil {
		// Container has already been removed.
		if errdefs.IsNotFound(err) {
			h.logger.Debug("attempted to stop nonexistent container")
			return nil
		}
		// Container has already been stopped.
		if errdefs.IsNotModified(err) {
			h.logger.Debug("attempted to stop an not-running container")
			return nil
		}

		h.logger.Error("failed to stop container", "error", err)
		return fmt.Errorf("Failed to stop container %s: %s", h.containerID, err)
	}

	h.logger.Info("stopped container")
	return nil
}

func (h *taskHandle) shutdownLogger() {
	if h.dlogger == nil {
		return
	}

	if err := h.dlogger.Stop(); err != nil {
		h.logger.Error("failed to stop docker logger process during StopTask",
			"error", err, "logger_pid", h.dloggerPluginClient.ReattachConfig().Pid)
	}
	h.dloggerPluginClient.Kill()
}

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Handle SIGTERM properly in the application or add a trap so the container exits promptly on stop.
2. Check the Docker daemon logs and restart Docker if it is wedged, then verify the container is gone ('docker rm -f <id>').
3. Force-remove the orphaned container manually, then re-run 'nomad alloc stop'.
4. Increase the task's kill_timeout so stop has more time before failing.

Example fix

// before: app ignores SIGTERM
kill_timeout = "5s"
// after
kill_timeout = "30s"  // plus a SIGTERM handler in the app

// give apps a workable SIGTERM window in the job spec
// kill_timeout = "30s"

err := client.Nodes.StopAlloc(ctx, allocID, nil)
if err != nil && strings.Contains(err.Error(), "Failed to stop container") {
    // container may still exist; force cleanup path
    log.Printf("stop failed, checking container state: %v", err)
    // verify alloc state / force-remove orphan via host tooling
}

When it happens

Trigger: h.dockerClient.ContainerStop(ctx, h.containerID, ...) returns a daemon error: container already removed, daemon down, timeout reached while the container ignored SIGTERM and stop force-failed, or storage/driver-level errors during shutdown.

Common situations: App ignoring SIGTERM so stop times out; Docker daemon crash during shutdown; overlay2/storage errors; container was deleted concurrently by another tool; host under heavy load slowing container teardown.

Related errors

• Failed to signal container %q while killing: %v
• Malformed hard ulimit %v: %v
• cannot apply %q configuration
• failed to create exec object: %v
• failed to attach to exec object: %w

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/0203fb0b374bb81c. Report an issue: GitHub.