{"record":{"id":"26ca4fe83bc3345d","repo":"hashicorp/nomad","slug":"failed-to-remove-pause-container-w","errorCode":null,"errorMessage":"failed to remove pause container: %w","messagePattern":"failed to remove pause container: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"drivers/docker/network.go","lineNumber":138,"sourceCode":"\n\t// no longer tracking this pause container; even if we fail here we should\n\t// let the background reconciliation keep trying\n\td.pauseContainers.remove(id)\n\n\tdockerClient, err := d.getDockerClient()\n\tif err != nil {\n\t\treturn fmt.Errorf(\"failed to connect to docker daemon: %s\", err)\n\t}\n\n\t// this is the pause container, just kill it fast\n\tif _, err := dockerClient.ContainerStop(d.ctx, id, mclient.ContainerStopOptions{Timeout: new(1)}); err != nil {\n\t\td.logger.Warn(\"failed to stop pause container\", \"id\", id, \"error\", err)\n\t}\n\n\tif _, err := dockerClient.ContainerRemove(d.ctx, id, mclient.ContainerRemoveOptions{\n\t\tForce: true,\n\t}); err != nil {\n\t\treturn fmt.Errorf(\"failed to remove pause container: %w\", err)\n\t}\n\n\tif d.config.GC.Image {\n\n\t\t// The Docker image ID is needed in order to correctly update the image\n\t\t// reference count. Any error finding this, however, should not result\n\t\t// in an error shutting down the allocrunner.\n\t\tdockerImage, err := dockerClient.ImageInspect(d.ctx, d.config.InfraImage)\n\t\tif err != nil {\n\t\t\td.logger.Warn(\"InspectImage failed for infra_image container destroy\",\n\t\t\t\t\"image\", d.config.InfraImage, \"error\", err)\n\t\t\treturn nil\n\t\t}\n\t\td.coordinator.RemoveImage(dockerImage.ID, allocID)\n\t}\n\n\treturn nil\n}","sourceCodeStart":120,"sourceCodeEnd":156,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/drivers/docker/network.go#L120-L156","documentation":"After stopping the pause container, DestroyNetwork force-removes it with ContainerRemove. If the Docker API remove call fails, the error is wrapped as 'failed to remove pause container'. The pause container tracking entry is already dropped, so reconciliation keeps trying even on failure.","triggerScenarios":"dockerClient.ContainerRemove(ctx, id, {Force:true}) returns an error other than success: container already removed by another actor, daemon connectivity dropped mid-call, or driver-level internal error (e.g. 'driver must be resumed' from an unexpected daemon state).","commonSituations":"Concurrent GC removing the same pause container; Docker daemon restarting during teardown; stale container state after daemon crash; host under heavy load causing API timeouts.","solutions":["Retry the destroy (background reconciliation does this automatically)","Check the container ID with docker ps -a and remove manually if orphaned: docker rm -f <id>","Verify the Docker daemon is healthy (docker info) and logs for internal errors","Restart the Nomad client task runner if its Docker client state is inconsistent with the daemon"],"exampleFix":"# before\ndocker ps -a | grep pause  # orphaned container remains\n# after\ndocker rm -f <pause-container-id>","handlingStrategy":"retry","validationCode":"// check the container exists before forcing removal\nif _, err := cli.ContainerInspect(ctx, pauseID); err != nil {\n    // not found: nothing to remove, treat as success\n    return nil\n}","typeGuard":"func isPauseRemoveErr(err error) bool {\n    return err != nil && strings.Contains(err.Error(), \"failed to remove pause container\")\n}","tryCatchPattern":"err := driver.DestroyNetwork(ctx, id)\nif isPauseRemoveErr(err) {\n    // reconciliation keeps trying; optionally force-remove manually\n    exec.Command(\"docker\", \"rm\", \"-f\", id).Run()\n}","preventionTips":["Avoid concurrent GC of the same allocation's network","Check daemon logs for internal driver errors on repeat failures","Treat NotFound removal errors as success in your own wrappers","Restart the client runner if its state diverges from the daemon"],"tags":["docker","network","cleanup","container-remove"],"backgroundTag":"docker-daemon-unreachable","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}