{"record":{"id":"5ab51430559568dc","repo":"hashicorp/nomad","slug":"failed-to-purge-container-s-s","errorCode":null,"errorMessage":"Failed to purge container %s: %s","messagePattern":"Failed to purge container (.+?): (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"drivers/docker/driver.go","lineNumber":560,"sourceCode":"\n\t\tcontainer, err := d.containerByName(config.Name)\n\t\tif err != nil {\n\t\t\treturn nil, err\n\t\t}\n\n\t\tif container != nil && container.Container.State.Running {\n\t\t\treturn container, nil\n\t\t}\n\n\t\t// Purge conflicting container if found.\n\t\t// If container is nil here, the conflicting container was\n\t\t// deleted in our check here, so retry again.\n\t\tif container != nil {\n\t\t\t// Delete matching containers\n\t\t\t_, err = client.ContainerRemove(d.ctx, container.Container.ID, mclient.ContainerRemoveOptions{Force: true})\n\t\t\tif err != nil {\n\t\t\t\td.logger.Error(\"failed to purge container\", \"container_id\", container.Container.ID)\n\t\t\t\treturn nil, recoverableErrTimeouts(fmt.Errorf(\"Failed to purge container %s: %s\", container.Container.ID, err))\n\t\t\t} else {\n\t\t\t\td.logger.Info(\"purged container\", \"container_id\", container.Container.ID)\n\t\t\t}\n\t\t}\n\n\t\tif attempted < d.config.ContainerExistsAttempts {\n\t\t\tattempted++\n\t\t\tbackoff = helper.Backoff(50*time.Millisecond, time.Minute, attempted)\n\t\t\ttime.Sleep(backoff)\n\t\t\tgoto CREATE\n\t\t}\n\n\t} else if errdefs.IsNotFound(createErr) {\n\t\t// There is still a very small chance this is possible even with the\n\t\t// coordinator so retry.\n\t\treturn nil, nstructs.NewRecoverableError(createErr, true)\n\t} else if isDockerTransientError(createErr) && attempted < 5 {\n\t\tattempted++","sourceCodeStart":542,"sourceCodeEnd":578,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/drivers/docker/driver.go#L542-L578","documentation":"Inside createContainer, when container creation hits a name conflict (errdefs.IsConflict), the driver looks up the stale container by name and tries to force-remove ('purge') it before retrying. This error is thrown when that force removal of the conflicting container fails, wrapped by recoverableErrTimeouts so timeouts are treated as recoverable. The stale container remains and blocks creating a container with the same name.","triggerScenarios":"A container with the same Name already exists from a previous task run, the lookup (containerByName) finds it not Running, and client.ContainerRemove(..., Force: true) errors - e.g. daemon timeout, container still tearing down, or daemon unreachable.","commonSituations":"Leftover containers from crashed previous allocations; two agents/nomad clients sharing one Docker daemon and fighting over the same container name; container in 'removal in progress' (Dead) state racing the remove; daemon stalled under load causing remove timeouts; devicemapper leftovers blocking removal.","solutions":["Wait and retry - the error is wrapped as recoverable on timeouts and createContainer retries with backoff up to ContainerExistsAttempts","Manually purge the stale container: docker rm -f <container_id> from the error message","Ensure only one orchestrator client manages this Docker daemon to avoid name conflicts","Increase ContainerExistsAttempts / check daemon responsiveness if removals repeatedly time out"],"exampleFix":"// before\n$ docker ps -a --filter name=<task-name>   # stale container blocks create\n// after\n$ docker rm -f <container_id>\n$ # or prevent leftovers: ensure previous allocs are cleaned before reusing same host/name","handlingStrategy":"retry","validationCode":"// Before task start, purge stale containers with the task's name pattern yourself:\nnames := []string{\"/\" + expectedContainerName}\nlist, err := cli.ContainerList(ctx, container.ListOptions{All: true})\nif err == nil {\n    for _, c := range list {\n        for _, n := range c.Names {\n            if strings.TrimPrefix(n, \"/\") == expectedContainerName {\n                _ = cli.ContainerRemove(ctx, c.ID, container.RemoveOptions{Force: true})\n            }\n        }\n    }\n}\n_ = names","typeGuard":"func isPurgeConflictErr(err error) bool {\n    return strings.Contains(err.Error(), \"Failed to purge container\")\n}","tryCatchPattern":"_, err := driver.StartTask(ctx, cfg)\nif err != nil && strings.Contains(err.Error(), \"Failed to purge container\") {\n    id := extractBetween(err.Error(), \"Failed to purge container \", \":\")\n    _ = exec.Command(\"docker\", \"rm\", \"-f\", id).Run() // clear stale container\n    time.Sleep(backoff) // createContainer also retries internally with backoff\n    return driver.StartTask(ctx, cfg)\n}","preventionTips":["Never run two orchestrator clients against the same Docker daemon with overlapping task names","Ensure allocation GC/cleanup completes before reusing the same container name on a host","Give the daemon headroom; remove timeouts are the most common purge failure under load","Use overlay2 storage driver and current Docker releases to avoid unremovable container leftovers"],"tags":["docker","container-name-conflict","cleanup"],"backgroundTag":"container-name-conflict","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}