{"record":{"id":"86d2f7e3fdbdebc1","repo":"hashicorp/nomad","slug":"failed-to-inspect-container-s-s","errorCode":null,"errorMessage":"Failed to inspect container %s: %s","messagePattern":"Failed to inspect container (.+?): (.+?)","errorType":"error_code","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"drivers/docker/driver.go","lineNumber":1722,"sourceCode":"OUTER:\n\tfor _, shimContainer = range containers.Items {\n\t\td.logger.Trace(\"listed container\", \"names\", hclog.Fmt(\"%+v\", shimContainer.Names))\n\t\tfor _, name := range shimContainer.Names {\n\t\t\tif name == containerName {\n\t\t\t\td.logger.Trace(\"Found container\",\n\t\t\t\t\t\"container_name\", containerName, \"container_id\", shimContainer.ID)\n\t\t\t\tfound = true\n\t\t\t\tbreak OUTER\n\t\t\t}\n\t\t}\n\t}\n\tif !found {\n\t\treturn nil, nil\n\t}\n\n\tcontainer, err := dockerClient.ContainerInspect(d.ctx, shimContainer.ID, mclient.ContainerInspectOptions{})\n\tif err != nil {\n\t\terr = fmt.Errorf(\"Failed to inspect container %s: %s\", shimContainer.ID, err)\n\n\t\t// This error is always recoverable as it could\n\t\t// be caused by races between listing\n\t\t// containers and this container being removed.\n\t\t// See #2802\n\t\treturn nil, nstructs.NewRecoverableError(err, true)\n\t}\n\treturn &container, nil\n}\n\n// validateCommand validates that the command only has a single value and\n// returns a user friendly error message telling them to use the passed\n// argField.\nfunc validateCommand(command, argField string) error {\n\ttrimmed := strings.TrimSpace(command)\n\tif len(trimmed) == 0 {\n\t\treturn fmt.Errorf(\"command empty: %q\", command)\n\t}","sourceCodeStart":1704,"sourceCodeEnd":1740,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/drivers/docker/driver.go#L1704-L1740","documentation":"After listing containers, containerByName inspects the matched container to get its full state. If ContainerInspect fails — most often because the container was removed between the list and the inspect (a race) — the error is wrapped in a RecoverableError(true) so Nomad retries.","triggerScenarios":"A race between ContainerList and ContainerInspect where the container is garbage-collected or stopped out of band (see Nomad #2802); or transient Docker API failures.","commonSituations":"Host under heavy container churn, out-of-band `docker rm`, Docker daemon GC, or external tooling (watchtower, kubelet-on-same-host) removing containers.","solutions":["Nothing to fix in the job — Nomad treats this as recoverable and retries; let the retry succeed.","Check for external tooling deleting containers (watchtower, cron docker rm) and exclude Nomad containers.","Verify no concurrent Docker daemon cleanup/GC is racing with Nomad on the host."],"exampleFix":null,"handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"if err != nil {\n    if nstructs.IsRecoverable(err) {\n        time.Sleep(500 * time.Millisecond)\n        return containerByName(name) // safe to retry: list-vs-inspect race\n    }\n    return nil, err\n}","preventionTips":["Disable external container-removal tooling (watchtower, cron docker rm) on Nomad clients","Accept that this race is expected under churn; rely on Nomad's recoverable-error retry","Keep the Docker daemon from aggressive GC of running containers"],"tags":["docker","nomad","race-condition","recoverable"],"backgroundTag":"docker-container-inspect-race","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}