hashicorp/nomad · error

failed to remove container %s: %v

Error message

failed to remove container %s: %v

What it means

After container creation failed, StartTask attempts to force-remove the partially created container to avoid leaking it; this error is thrown when that cleanup removal itself fails. The original create error is then masked by this removal failure, so both the create error and remove error are relevant. It means a broken container object still exists on the daemon and could not be deleted.

Source

Thrown at drivers/docker/driver.go:401

		}
	}

	containerCfg, err := d.createContainerConfig(cfg, &driverConfig, driverConfig.Image)
	if err != nil {
		d.logger.Error("failed to create container configuration", "image_name", driverConfig.Image,
			"image_id", id, "error", err)
		return nil, nil, fmt.Errorf("Failed to create container configuration for image %q (%q): %v", driverConfig.Image, id, err)
	}

	startAttempts := 0
CREATE:
	container, err := d.createContainer(dockerClient, containerCfg, driverConfig.Image)
	if err != nil {
		d.logger.Error("failed to create container", "error", err)
		if container != nil {
			_, removeErr := dockerClient.ContainerRemove(d.ctx, container.Container.ID, mclient.ContainerRemoveOptions{Force: true})
			if removeErr != nil {
				return nil, nil, fmt.Errorf("failed to remove container %s: %v", container.Container.ID, removeErr)
			}
		}
		return nil, nil, nstructs.WrapRecoverable(fmt.Sprintf("failed to create container: %v", err), err)
	}

	d.logger.Info("created container", "container_id", container.Container.ID)

	if !container.Container.State.Running {
		// Start the container
		if err := d.startContainer(*container); err != nil {
			d.logger.Error("failed to start container", "container_id", container.Container.ID, "error", err)
			_, _ = dockerClient.ContainerRemove(d.ctx, container.Container.ID, mclient.ContainerRemoveOptions{Force: true})
			// Some sort of docker race bug, recreating the container usually works
			if errdefs.IsConflict(err) && startAttempts < 5 {
				startAttempts++
				d.logger.Debug("reattempting container create/start sequence", "attempt", startAttempts, "container_id", id)
				goto CREATE
			}

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Retry StartTask once the Docker daemon is confirmed healthy (docker info works)
2. Manually run `docker rm -f <container_id>` shown in the error to clear the leaked container
3. Check daemon logs and storage-driver health if removal repeatedly fails (devicemapper/aufs leftovers)
4. Upgrade Docker if the removal fails due to known storage-driver bugs with force removal

Example fix

// before
$ docker ps -a | grep <id>   # container still present, agent retries fail
// after
$ docker rm -f <container_id> && docker info   # then restart the allocation

// Check daemon health before starting tasks so create/remove calls don't fail mid-flight:
cli, err := client.NewClientWithOpts(client.FromEnv)
if err == nil {
    ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
    defer cancel()
    if _, err := cli.Info(ctx); err != nil {
        return fmt.Errorf("docker daemon unhealthy: %w", err)
    }
}

_, _, err := driver.StartTask(ctx, cfg)
if err != nil {
    var rec recoverableError
    if strings.Contains(err.Error(), "failed to remove container") {
        // daemon-side cleanup failed; clear the container then retry the task
        id := extractContainerID(err.Error())
        _ = exec.Command("docker", "rm", "-f", id).Run()
        return retryStartTask(cfg)
    }
    return err
}

When it happens

Trigger: d.createContainer returns a non-nil container along with an error, and the subsequent dockerClient.ContainerRemove(..., Force: true) on container.Container.ID returns an error (e.g. daemon connection lost mid-operation, remove raced with daemon GC, or device/resource busy).

Common situations: Docker daemon became unreachable between create and remove; container was already removed by another process causing a race; container pinned by a stopped devmapper/shim on older kernels; storage driver (devicemapper/aufs) left the container in a state blocking force removal.

Related errors

• failed to setup replacement docker logger: %v
• Failed to purge container %s: %s
• failed to remove pause container: %w
• running container as ContainerAdmin is unsafe; change the co
• error decoding stats data: no reader body

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/3db867a72c16634a. Report an issue: GitHub.