hashicorp/nomad · error

failed to remove pause container: %w

Error message

failed to remove pause container: %w

What it means

After stopping the pause container, DestroyNetwork force-removes it with ContainerRemove. If the Docker API remove call fails, the error is wrapped as 'failed to remove pause container'. The pause container tracking entry is already dropped, so reconciliation keeps trying even on failure.

Source

Thrown at drivers/docker/network.go:138

	// no longer tracking this pause container; even if we fail here we should
	// let the background reconciliation keep trying
	d.pauseContainers.remove(id)

	dockerClient, err := d.getDockerClient()
	if err != nil {
		return fmt.Errorf("failed to connect to docker daemon: %s", err)
	}

	// this is the pause container, just kill it fast
	if _, err := dockerClient.ContainerStop(d.ctx, id, mclient.ContainerStopOptions{Timeout: new(1)}); err != nil {
		d.logger.Warn("failed to stop pause container", "id", id, "error", err)
	}

	if _, err := dockerClient.ContainerRemove(d.ctx, id, mclient.ContainerRemoveOptions{
		Force: true,
	}); err != nil {
		return fmt.Errorf("failed to remove pause container: %w", err)
	}

	if d.config.GC.Image {

		// The Docker image ID is needed in order to correctly update the image
		// reference count. Any error finding this, however, should not result
		// in an error shutting down the allocrunner.
		dockerImage, err := dockerClient.ImageInspect(d.ctx, d.config.InfraImage)
		if err != nil {
			d.logger.Warn("InspectImage failed for infra_image container destroy",
				"image", d.config.InfraImage, "error", err)
			return nil
		}
		d.coordinator.RemoveImage(dockerImage.ID, allocID)
	}

	return nil
}

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Retry the destroy (background reconciliation does this automatically)
2. Check the container ID with docker ps -a and remove manually if orphaned: docker rm -f <id>
3. Verify the Docker daemon is healthy (docker info) and logs for internal errors
4. Restart the Nomad client task runner if its Docker client state is inconsistent with the daemon

Example fix

# before
docker ps -a | grep pause  # orphaned container remains
# after
docker rm -f <pause-container-id>

// check the container exists before forcing removal
if _, err := cli.ContainerInspect(ctx, pauseID); err != nil {
    // not found: nothing to remove, treat as success
    return nil
}

func isPauseRemoveErr(err error) bool {
    return err != nil && strings.Contains(err.Error(), "failed to remove pause container")
}

err := driver.DestroyNetwork(ctx, id)
if isPauseRemoveErr(err) {
    // reconciliation keeps trying; optionally force-remove manually
    exec.Command("docker", "rm", "-f", id).Run()
}

When it happens

Trigger: dockerClient.ContainerRemove(ctx, id, {Force:true}) returns an error other than success: container already removed by another actor, daemon connectivity dropped mid-call, or driver-level internal error (e.g. 'driver must be resumed' from an unexpected daemon state).

Common situations: Concurrent GC removing the same pause container; Docker daemon restarting during teardown; stale container state after daemon crash; host under heavy load causing API timeouts.

Related errors

• failed to fetch docker daemon info: %v
• failed to setup replacement docker logger: %v
• failed to remove container %s: %v
• Failed to purge container %s: %s
• Port %q not found, check network block

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/26ca4fe83bc3345d. Report an issue: GitHub.