hashicorp/nomad · error

Failed to purge container %s: %s

Error message

Failed to purge container %s: %s

What it means

Inside createContainer, when container creation hits a name conflict (errdefs.IsConflict), the driver looks up the stale container by name and tries to force-remove ('purge') it before retrying. This error is thrown when that force removal of the conflicting container fails, wrapped by recoverableErrTimeouts so timeouts are treated as recoverable. The stale container remains and blocks creating a container with the same name.

Source

Thrown at drivers/docker/driver.go:560

		container, err := d.containerByName(config.Name)
		if err != nil {
			return nil, err
		}

		if container != nil && container.Container.State.Running {
			return container, nil
		}

		// Purge conflicting container if found.
		// If container is nil here, the conflicting container was
		// deleted in our check here, so retry again.
		if container != nil {
			// Delete matching containers
			_, err = client.ContainerRemove(d.ctx, container.Container.ID, mclient.ContainerRemoveOptions{Force: true})
			if err != nil {
				d.logger.Error("failed to purge container", "container_id", container.Container.ID)
				return nil, recoverableErrTimeouts(fmt.Errorf("Failed to purge container %s: %s", container.Container.ID, err))
			} else {
				d.logger.Info("purged container", "container_id", container.Container.ID)
			}
		}

		if attempted < d.config.ContainerExistsAttempts {
			attempted++
			backoff = helper.Backoff(50*time.Millisecond, time.Minute, attempted)
			time.Sleep(backoff)
			goto CREATE
		}

	} else if errdefs.IsNotFound(createErr) {
		// There is still a very small chance this is possible even with the
		// coordinator so retry.
		return nil, nstructs.NewRecoverableError(createErr, true)
	} else if isDockerTransientError(createErr) && attempted < 5 {
		attempted++

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Wait and retry - the error is wrapped as recoverable on timeouts and createContainer retries with backoff up to ContainerExistsAttempts
2. Manually purge the stale container: docker rm -f <container_id> from the error message
3. Ensure only one orchestrator client manages this Docker daemon to avoid name conflicts
4. Increase ContainerExistsAttempts / check daemon responsiveness if removals repeatedly time out

Example fix

// before
$ docker ps -a --filter name=<task-name>   # stale container blocks create
// after
$ docker rm -f <container_id>
$ # or prevent leftovers: ensure previous allocs are cleaned before reusing same host/name

// Before task start, purge stale containers with the task's name pattern yourself:
names := []string{"/" + expectedContainerName}
list, err := cli.ContainerList(ctx, container.ListOptions{All: true})
if err == nil {
    for _, c := range list {
        for _, n := range c.Names {
            if strings.TrimPrefix(n, "/") == expectedContainerName {
                _ = cli.ContainerRemove(ctx, c.ID, container.RemoveOptions{Force: true})
            }
        }
    }
}
_ = names

func isPurgeConflictErr(err error) bool {
    return strings.Contains(err.Error(), "Failed to purge container")
}

_, err := driver.StartTask(ctx, cfg)
if err != nil && strings.Contains(err.Error(), "Failed to purge container") {
    id := extractBetween(err.Error(), "Failed to purge container ", ":")
    _ = exec.Command("docker", "rm", "-f", id).Run() // clear stale container
    time.Sleep(backoff) // createContainer also retries internally with backoff
    return driver.StartTask(ctx, cfg)
}

When it happens

Trigger: A container with the same Name already exists from a previous task run, the lookup (containerByName) finds it not Running, and client.ContainerRemove(..., Force: true) errors - e.g. daemon timeout, container still tearing down, or daemon unreachable.

Common situations: Leftover containers from crashed previous allocations; two agents/nomad clients sharing one Docker daemon and fighting over the same container name; container in 'removal in progress' (Dead) state racing the remove; daemon stalled under load causing remove timeouts; devicemapper leftovers blocking removal.

Related errors

• failed to setup replacement docker logger: %v
• failed to remove container %s: %v
• failed to remove pause container: %w
• running container as ContainerAdmin is unsafe; change the co
• error decoding stats data: no reader body

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/5ab51430559568dc. Report an issue: GitHub.