{"record":{"id":"794e48feb73dfc5f","repo":"hashicorp/nomad","slug":"failed-to-inspect-container-q-v","errorCode":null,"errorMessage":"failed to inspect container %q: %v","messagePattern":"failed to inspect container %q: (.+?)","errorType":"error_code","errorClass":null,"httpStatus":null,"severity":"error","filePath":"drivers/docker/driver.go","lineNumber":1853,"sourceCode":"\td.coordinator.RemoveImage(handle.containerImage, handle.task.ID)\n\n\treturn nil\n}\n\nfunc (d *Driver) InspectTask(taskID string) (*drivers.TaskStatus, error) {\n\th, ok := d.tasks.Get(taskID)\n\tif !ok {\n\t\treturn nil, drivers.ErrTaskNotFound\n\t}\n\n\tdockerClient, err := d.getDockerClient()\n\tif err != nil {\n\t\treturn nil, err\n\t}\n\n\tcontainer, err := dockerClient.ContainerInspect(d.ctx, h.containerID, mclient.ContainerInspectOptions{})\n\tif err != nil {\n\t\treturn nil, fmt.Errorf(\"failed to inspect container %q: %v\", h.containerID, err)\n\t}\n\n\tstarted, _ := time.Parse(time.RFC3339, container.Container.State.StartedAt)\n\tcompleted, _ := time.Parse(time.RFC3339, container.Container.State.FinishedAt)\n\n\tstatus := &drivers.TaskStatus{\n\t\tID:          h.task.ID,\n\t\tName:        h.task.Name,\n\t\tStartedAt:   started,\n\t\tCompletedAt: completed,\n\t\tDriverAttributes: map[string]string{\n\t\t\t\"container_id\": container.Container.ID,\n\t\t},\n\t\tNetworkOverride: h.net,\n\t\tExitResult:      h.ExitResult(),\n\t}\n\n\tstatus.State = drivers.TaskStateUnknown","sourceCodeStart":1835,"sourceCodeEnd":1871,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/drivers/docker/driver.go#L1835-L1871","documentation":"This error wraps a failure from the Docker Engine API `ContainerInspect` call while building a task's status snapshot in the Nomad Docker driver. The driver inspects the container to read State.StartedAt/FinishedAt and derive task status; if the Docker daemon cannot return the container metadata, the underlying error is wrapped with the container ID for context. It means the driver has a handle for the task but the container backing it is no longer visible or the API call failed.","triggerScenarios":"Calling the driver's InsTaskStatus path when dockerClient.ContainerInspect(ctx, h.containerID, ...) returns an error — typically because the container was removed concurrently, the container ID is stale, or the Docker daemon is unreachable.","commonSituations":"Container exited and was auto-removed (or garbage collected) before status was inspected; Docker daemon restart/OOM; task handle outlived the container after a node event; network interruption between Nomad and dockerd.","solutions":["Check the container still exists with `docker ps -a | grep <containerID>`; if removed, re-run the task or clear the stale handle.","Verify the Docker daemon is healthy: `docker info` and `systemctl status docker`.","Inspect client/daemon connectivity (DOCKER_HOST, socket permissions, remote API reachability).","Upgrade the Docker client library/daemon if ContainerInspect intermittently fails after daemon restarts."],"exampleFix":"// before\ncontainer, err := dockerClient.ContainerInspect(d.ctx, h.containerID, mclient.ContainerInspectOptions{})\nif err != nil {\n    return nil, fmt.Errorf(\"failed to inspect container %q: %v\", h.containerID, err)\n}\n// after\nccontainer, err := dockerClient.ContainerInspect(d.ctx, h.containerID, mclient.ContainerInspectOptions{})\nif err != nil {\n    if errdefs.IsNotFound(err) {\n        return nil, fmt.Errorf(\"container %q no longer exists: %w\", h.containerID, err)\n    }\n    return nil, fmt.Errorf(\"failed to inspect container %q: %v\", h.containerID, err)\n}","handlingStrategy":"try-catch","validationCode":null,"typeGuard":"func isContainerNotFound(err error) bool { return errdefs.IsNotFound(err) || strings.Contains(err.Error(), \"No such container\") }","tryCatchPattern":"status, err := driver.InspectTask(taskID)\nif err != nil {\n    if isContainerNotFound(err) {\n        // treat task as dead; cleanup handle\n        return drivers.ErrTaskNotFound\n    }\n    return fmt.Errorf(\"status inspection failed, check dockerd health: %w\", err)\n}","preventionTips":["Avoid aggressive `--rm`/auto-remove settings that race status inspection.","Monitor dockerd health and restart policies on the node.","Keep Docker client and daemon versions aligned.","Log the container ID from the error to correlate with daemon logs."],"tags":["docker","container-inspect","driver"],"backgroundTag":"container-not-found","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}