{"record":{"id":"e21b20fefd934d93","repo":"hashicorp/nomad","slug":"failed-to-inspect-container-for-id-q-v","errorCode":null,"errorMessage":"failed to inspect container for id %q: %v","messagePattern":"failed to inspect container for id %q: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"drivers/docker/driver.go","lineNumber":270,"sourceCode":"\n\tdockerClient, err := d.getDockerClient()\n\tif err != nil {\n\t\treturn fmt.Errorf(\"failed to get docker client: %w\", err)\n\t}\n\n\tdockerInfo, err := dockerClient.Info(d.ctx, mclient.InfoOptions{})\n\tif err != nil {\n\t\treturn fmt.Errorf(\"failed to fetch docker daemon info: %v\", err)\n\t}\n\n\tinfinityClient, err := d.getInfinityClient()\n\tif err != nil {\n\t\treturn fmt.Errorf(\"failed to get docker long operations client: %w\", err)\n\t}\n\n\tcontainer, err := dockerClient.ContainerInspect(d.ctx, handleState.ContainerID, mclient.ContainerInspectOptions{})\n\tif err != nil {\n\t\treturn fmt.Errorf(\"failed to inspect container for id %q: %v\", handleState.ContainerID, err)\n\t}\n\n\th := &taskHandle{\n\t\tdockerClient:            dockerClient,\n\t\tdockerCGroupDriver:      dockerInfo.Info.CgroupDriver,\n\t\tinfinityClient:          infinityClient,\n\t\tlogger:                  d.logger.With(\"container_id\", container.Container.ID),\n\t\ttask:                    handle.Config,\n\t\tcontainerID:             container.Container.ID,\n\t\tcontainerCgroup:         string(container.Container.HostConfig.Cgroup),\n\t\tcontainerImage:          container.Container.Image,\n\t\tdoneCh:                  make(chan bool),\n\t\twaitCh:                  make(chan struct{}),\n\t\tremoveContainerOnExit:   d.config.GC.Container,\n\t\tnet:                     handleState.DriverNetwork,\n\t\tdisableCpusetManagement: d.config.disableCpusetManagement,\n\t}\n","sourceCodeStart":252,"sourceCodeEnd":288,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/drivers/docker/driver.go#L252-L288","documentation":"RecoverTask inspects the persisted container (handleState.ContainerID) to rebuild the task handle. If the Docker API inspect call fails — most commonly because the container no longer exists — recovery of that task is impossible and this error names the offending container ID.","triggerScenarios":"Container was removed manually (`docker rm`) or by a GC/cleanup script between the client crash and recovery; container ID in the persisted state is stale or corrupted; daemon was restarted with a different data root; inspect times out or the daemon errors.","commonSituations":"Operator cleaned up 'orphaned' containers while the orchestrator was down; Docker upgraded/reinstalled losing containers; state file references a container from a different daemon (DOCKER_HOST changed); disk pressure cleanup daemon deleting containers.","solutions":["Check whether the container exists: `docker inspect <containerID>`. If it is gone, the task cannot be recovered — kill/forget the task handle instead of retrying.","Verify DOCKER_HOST/endpoint points to the same daemon that originally ran the container.","If containers were GC'd, disable aggressive container cleanup (d.config.GC.Container) or cleanup scripts on hosts running recoverable tasks.","If the ID is stale in the state store, clear the corrupted task state so the task can be rescheduled fresh."],"exampleFix":"// before\ncontainer, err := dockerClient.ContainerInspect(d.ctx, handleState.ContainerID, mclient.ContainerInspectOptions{})\nif err != nil {\n\treturn fmt.Errorf(\"failed to inspect container for id %q: %v\", handleState.ContainerID, err)\n}\n// after\ncontainer, err := dockerClient.ContainerInspect(d.ctx, handleState.ContainerID, mclient.ContainerInspectOptions{})\nif err != nil {\n\tif mclient.IsErrNotFound(err) {\n\t\td.logger.Warn(\"container gone; cannot recover task\", \"container_id\", handleState.ContainerID)\n\t\treturn drivers.ErrTaskNotFound // signal unrecoverable instead of generic error\n\t}\n\treturn fmt.Errorf(\"failed to inspect container for id %q: %w\", handleState.ContainerID, err)\n}","handlingStrategy":"try-catch","validationCode":"// check container existence before attempting recovery\nfunc containerExists(ctx context.Context, cli *client.Client, id string) (bool, error) {\n\t_, err := cli.ContainerInspect(ctx, id)\n\tif err != nil {\n\t\tif errdefs.IsNotFound(err) {\n\t\t\treturn false, nil\n\t\t}\n\t\treturn false, err\n\t}\n\treturn true, nil\n}","typeGuard":"func isContainerNotFound(err error) bool {\n\treturn errdefs.IsNotFound(err) // or strings.Contains(err.Error(), \"No such container\")\n}","tryCatchPattern":"// Go: treat not-found as unrecoverable, other errors as retryable\nif err := driver.RecoverTask(handle); err != nil {\n\tif strings.Contains(err.Error(), \"failed to inspect container\") {\n\t\tif isContainerNotFound(errors.Unwrap(err)) {\n\t\t\t// drop the handle / reschedule the task; do not retry\n\t\t} else {\n\t\t\t// transient daemon error: retry with backoff\n\t\t}\n\t}\n}","preventionTips":["Never manually `docker rm` containers on hosts running orchestrated tasks.","Exclude agent state/container dirs from GC scripts and disk cleaners.","Keep DOCKER_HOST/data-root unchanged across docker upgrades.","Persist container IDs only against the daemon that created them."],"tags":["docker","container-missing","task-recovery","state"],"backgroundTag":"container-not-found","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}