{"record":{"id":"3db867a72c16634a","repo":"hashicorp/nomad","slug":"failed-to-remove-container-s-v","errorCode":null,"errorMessage":"failed to remove container %s: %v","messagePattern":"failed to remove container (.+?): (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"drivers/docker/driver.go","lineNumber":401,"sourceCode":"\t\t}\n\t}\n\n\tcontainerCfg, err := d.createContainerConfig(cfg, &driverConfig, driverConfig.Image)\n\tif err != nil {\n\t\td.logger.Error(\"failed to create container configuration\", \"image_name\", driverConfig.Image,\n\t\t\t\"image_id\", id, \"error\", err)\n\t\treturn nil, nil, fmt.Errorf(\"Failed to create container configuration for image %q (%q): %v\", driverConfig.Image, id, err)\n\t}\n\n\tstartAttempts := 0\nCREATE:\n\tcontainer, err := d.createContainer(dockerClient, containerCfg, driverConfig.Image)\n\tif err != nil {\n\t\td.logger.Error(\"failed to create container\", \"error\", err)\n\t\tif container != nil {\n\t\t\t_, removeErr := dockerClient.ContainerRemove(d.ctx, container.Container.ID, mclient.ContainerRemoveOptions{Force: true})\n\t\t\tif removeErr != nil {\n\t\t\t\treturn nil, nil, fmt.Errorf(\"failed to remove container %s: %v\", container.Container.ID, removeErr)\n\t\t\t}\n\t\t}\n\t\treturn nil, nil, nstructs.WrapRecoverable(fmt.Sprintf(\"failed to create container: %v\", err), err)\n\t}\n\n\td.logger.Info(\"created container\", \"container_id\", container.Container.ID)\n\n\tif !container.Container.State.Running {\n\t\t// Start the container\n\t\tif err := d.startContainer(*container); err != nil {\n\t\t\td.logger.Error(\"failed to start container\", \"container_id\", container.Container.ID, \"error\", err)\n\t\t\t_, _ = dockerClient.ContainerRemove(d.ctx, container.Container.ID, mclient.ContainerRemoveOptions{Force: true})\n\t\t\t// Some sort of docker race bug, recreating the container usually works\n\t\t\tif errdefs.IsConflict(err) && startAttempts < 5 {\n\t\t\t\tstartAttempts++\n\t\t\t\td.logger.Debug(\"reattempting container create/start sequence\", \"attempt\", startAttempts, \"container_id\", id)\n\t\t\t\tgoto CREATE\n\t\t\t}","sourceCodeStart":383,"sourceCodeEnd":419,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/drivers/docker/driver.go#L383-L419","documentation":"After container creation failed, StartTask attempts to force-remove the partially created container to avoid leaking it; this error is thrown when that cleanup removal itself fails. The original create error is then masked by this removal failure, so both the create error and remove error are relevant. It means a broken container object still exists on the daemon and could not be deleted.","triggerScenarios":"d.createContainer returns a non-nil container along with an error, and the subsequent dockerClient.ContainerRemove(..., Force: true) on container.Container.ID returns an error (e.g. daemon connection lost mid-operation, remove raced with daemon GC, or device/resource busy).","commonSituations":"Docker daemon became unreachable between create and remove; container was already removed by another process causing a race; container pinned by a stopped devmapper/shim on older kernels; storage driver (devicemapper/aufs) left the container in a state blocking force removal.","solutions":["Retry StartTask once the Docker daemon is confirmed healthy (docker info works)","Manually run `docker rm -f <container_id>` shown in the error to clear the leaked container","Check daemon logs and storage-driver health if removal repeatedly fails (devicemapper/aufs leftovers)","Upgrade Docker if the removal fails due to known storage-driver bugs with force removal"],"exampleFix":"// before\n$ docker ps -a | grep <id>   # container still present, agent retries fail\n// after\n$ docker rm -f <container_id> && docker info   # then restart the allocation","handlingStrategy":"retry","validationCode":"// Check daemon health before starting tasks so create/remove calls don't fail mid-flight:\ncli, err := client.NewClientWithOpts(client.FromEnv)\nif err == nil {\n    ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)\n    defer cancel()\n    if _, err := cli.Info(ctx); err != nil {\n        return fmt.Errorf(\"docker daemon unhealthy: %w\", err)\n    }\n}","typeGuard":null,"tryCatchPattern":"_, _, err := driver.StartTask(ctx, cfg)\nif err != nil {\n    var rec recoverableError\n    if strings.Contains(err.Error(), \"failed to remove container\") {\n        // daemon-side cleanup failed; clear the container then retry the task\n        id := extractContainerID(err.Error())\n        _ = exec.Command(\"docker\", \"rm\", \"-f\", id).Run()\n        return retryStartTask(cfg)\n    }\n    return err\n}","preventionTips":["Monitor Docker daemon health (docker info in a periodic check) before/while scheduling tasks","Enable storage drivers with clean force-removal behavior (overlay2) and retire devicemapper/aufs","Avoid racing external container GC/reapers with the driver's own cleanup","Keep the agent on the same host as the daemon (local socket) to reduce mid-operation disconnects"],"tags":["docker","container-lifecycle","cleanup"],"backgroundTag":"container-remove-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T15:18:49.778Z"}