{"record":{"id":"a4a775745bc66a66","repo":"hashicorp/nomad","slug":"s-s-s","errorCode":null,"errorMessage":"%s %s: %s","messagePattern":"%s %s: %s","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"drivers/docker/driver.go","lineNumber":431,"sourceCode":"\t\t\t_, _ = dockerClient.ContainerRemove(d.ctx, container.Container.ID, mclient.ContainerRemoveOptions{Force: true})\n\t\t\t// Some sort of docker race bug, recreating the container usually works\n\t\t\tif errdefs.IsConflict(err) && startAttempts < 5 {\n\t\t\t\tstartAttempts++\n\t\t\t\td.logger.Debug(\"reattempting container create/start sequence\", \"attempt\", startAttempts, \"container_id\", id)\n\t\t\t\tgoto CREATE\n\t\t\t}\n\t\t\treturn nil, nil, nstructs.WrapRecoverable(fmt.Sprintf(\"Failed to start container %s: %s\", container.Container.ID, err), err)\n\t\t}\n\n\t\t// Inspect container to get all of the container metadata as much of the\n\t\t// metadata (eg networking) isn't populated until the container is started\n\t\trunningContainer, err := dockerClient.ContainerInspect(d.ctx, container.Container.ID, mclient.ContainerInspectOptions{})\n\t\tif err != nil {\n\t\t\t_, _ = dockerClient.ContainerRemove(d.ctx, container.Container.ID, mclient.ContainerRemoveOptions{Force: true})\n\t\t\tmsg := \"failed to inspect started container\"\n\t\t\td.logger.Error(msg, \"error\", err)\n\t\t\t_, _ = dockerClient.ContainerRemove(d.ctx, container.Container.ID, mclient.ContainerRemoveOptions{Force: true})\n\t\t\treturn nil, nil, nstructs.NewRecoverableError(fmt.Errorf(\"%s %s: %s\", msg, container.Container.ID, err), true)\n\t\t}\n\t\tcontainer = &runningContainer\n\t\td.logger.Info(\"started container\", \"container_id\", container.Container.ID)\n\t} else {\n\t\td.logger.Debug(\"re-attaching to container\", \"container_id\",\n\t\t\tcontainer.Container.ID, \"container_state\", container.Container.State.Status)\n\t}\n\n\tcollectingLogs := loggingIsEnabled(d.config, cfg)\n\n\tvar dlogger docklog.DockerLogger\n\tvar pluginClient *plugin.Client\n\n\tif collectingLogs {\n\t\tdlogger, pluginClient, err = d.setupNewDockerLogger(*container, cfg, time.Unix(0, 0))\n\t\tif err != nil {\n\t\t\td.logger.Error(\"an error occurred after container startup, terminating container\", \"container_id\", container.Container.ID)\n\t\t\t_, _ = dockerClient.ContainerRemove(d.ctx, container.Container.ID, mclient.ContainerRemoveOptions{Force: true})","sourceCodeStart":413,"sourceCodeEnd":449,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/drivers/docker/driver.go#L413-L449","documentation":"After the container was started, StartTask inspects it (ContainerInspect) to obtain fresh state; this error is thrown when the inspect call fails. The driver force-removes the container and returns a recoverable error, so the allocation can be retried on another placement or later. It generally means the daemon could not answer the inspect, typically because the daemon connection dropped or the container exited and was reaped between start and inspect.","triggerScenarios":"dockerClient.ContainerInspect on the just-started container returns an error: daemon connection reset/timeout, container no longer exists (exited instantly and removed), or daemon restart mid-startup.","commonSituations":"Docker daemon crashed or restarted right after container start; container's entrypoint fails immediately so the container disappears before inspect; network interruption between agent and remote Docker daemon (DOCKER_HOST over tcp); long daemon stalls exceeding the client timeout.","solutions":["Check `docker logs`/daemon logs for why the container vanished immediately (fast-exiting entrypoint)","Verify the Docker daemon is stable and reachable (docker info); fix daemon crashes before retrying","Let the orchestrator retry - the error is marked recoverable, so rescheduling may succeed","If over TCP, check network stability/timeouts between agent and the remote Docker host"],"exampleFix":"// before\nCMD [\"myapp\"]   # crashes instantly, container gone before inspect\n// after\nCMD [\"myapp\"] \n# plus verify locally: docker run --rm image myapp - stays up; fix app crash or missing env/config","handlingStrategy":"retry","validationCode":"// Verify the daemon is stable and the image entrypoint stays up before submitting:\nctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)\ndefer cancel()\nif _, err := dockerCli.Info(ctx); err != nil {\n    return fmt.Errorf(\"daemon unreachable, task start will likely fail inspect: %w\", err)\n}\nout, err := dockerCli.ContainerCreate(ctx, cfg, nil, nil, nil, \"smoke-test\")\n_ = out\nif err != nil { return err }","typeGuard":"func isRecoverableInspectErr(err error) bool {\n    return strings.Contains(err.Error(), \"failed to inspect started container\")\n}","tryCatchPattern":"h, err := driver.StartTask(ctx, cfg)\nif err != nil && strings.Contains(err.Error(), \"failed to inspect started container\") {\n    // driver already force-removed the container and marked this recoverable\n    time.Sleep(backoff)\n    h, err = driver.StartTask(ctx, cfg) // safe to retry placement\n}","preventionTips":["Fix application entrypoints that exit immediately (crash loops cause start->inspect races)","Use a local unix socket instead of TCP DOCKER_HOST to avoid network-induced inspect failures","Ensure the daemon isn't restarted/crash-looping (monitor dockerd and its journal logs)","Keep Docker client timeouts generous enough for loaded daemons"],"tags":["docker","container-inspect","recoverable"],"backgroundTag":"container-inspect-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}