{"record":{"id":"7b70b38724fd8a51","repo":"hashicorp/nomad","slug":"failed-to-inspect-container-state-v","errorCode":null,"errorMessage":"failed to inspect container state: %v","messagePattern":"failed to inspect container state: (.+?)","errorType":"error_code","errorClass":null,"httpStatus":null,"severity":"error","filePath":"drivers/docker/driver.go","lineNumber":1797,"sourceCode":"\nfunc (d *Driver) DestroyTask(taskID string, force bool) error {\n\th, ok := d.tasks.Get(taskID)\n\tif !ok {\n\t\treturn drivers.ErrTaskNotFound\n\t}\n\n\tdockerClient, err := d.getDockerClient()\n\tif err != nil {\n\t\treturn err\n\t}\n\n\tc, err := dockerClient.ContainerInspect(d.ctx, h.containerID, mclient.ContainerInspectOptions{})\n\tif err != nil {\n\t\tif errdefs.IsNotFound(err) {\n\t\t\th.logger.Info(\"container was removed out of band, will proceed with DestroyTask\",\n\t\t\t\t\"error\", err)\n\t\t} else {\n\t\t\treturn fmt.Errorf(\"failed to inspect container state: %v\", err)\n\t\t}\n\t} else {\n\t\tif c.Container.State.Running {\n\t\t\tif !force {\n\t\t\t\treturn fmt.Errorf(\"must call StopTask for the given task before Destroy or set force to true\")\n\t\t\t}\n\t\t\tif _, err := dockerClient.ContainerStop(d.ctx, h.containerID, mclient.ContainerStopOptions{Timeout: new(0)}); err != nil {\n\t\t\t\th.logger.Warn(\"failed to stop container during destroy\", \"error\", err)\n\t\t\t}\n\t\t}\n\n\t\tif h.removeContainerOnExit {\n\t\t\tif _, err := dockerClient.ContainerRemove(d.ctx, h.containerID, mclient.ContainerRemoveOptions{Force: true}); err != nil {\n\t\t\t\th.logger.Error(\"error removing container\", \"error\", err)\n\t\t\t}\n\t\t} else {\n\t\t\th.logger.Debug(\"not removing container due to config\")\n\t\t}","sourceCodeStart":1779,"sourceCodeEnd":1815,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/drivers/docker/driver.go#L1779-L1815","documentation":"DestroyTask inspects the container before removing it. If the inspect call fails with any error other than 'not found' (daemon down, timeout, permission), the destroy aborts with this error instead of proceeding, because Nomad cannot verify the container's state.","triggerScenarios":"Docker daemon unreachable or timing out during DestroyTask; transient API errors that are not errdefs.IsNotFound while tearing down an allocation.","commonSituations":"Docker daemon hung under load, socket permission changes, host under heavy I/O making the API time out during alloc GC or node drain.","solutions":["Verify the Docker daemon is healthy (`docker ps`, `systemctl status docker`) and retry the destroy/GC.","If the container is already gone but inspect errored transiently, the next GC pass will remove the leftover tracking state.","Check docker.sock permissions and docker logs for API errors; restart the daemon if wedged."],"exampleFix":"// before\nsudo docker ps   # hangs or errors\n// after\nsudo systemctl restart docker\nnomad node drain <node> -enable   # or let alloc GC retry","handlingStrategy":"retry","validationCode":"// preflight before destroy/GC: daemon must answer\ncmd := exec.Command(\"docker\", \"info\")\nif err := cmd.Run(); err != nil { log.Printf(\"docker daemon down; defer destroy\") }","typeGuard":null,"tryCatchPattern":"if err := destroy(); err != nil {\n    if strings.Contains(err.Error(), \"failed to inspect container state\") {\n        time.Sleep(retryBackoff)\n        return destroy() // transient daemon issue; retry after docker recovers\n    }\n    return err\n}","preventionTips":["Alert on Docker daemon health so destroys don't collide with outages","Ensure nomad user retains docker.sock permissions across daemon upgrades","Retry alloc GC rather than forcing removal when inspect fails"],"tags":["docker","nomad","container-inspect","daemon-unreachable"],"backgroundTag":"docker-daemon-unreachable","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}