{"record":{"id":"3818d1f817451a5c","repo":"hashicorp/nomad","slug":"failed-to-fetch-docker-daemon-info-v","errorCode":null,"errorMessage":"failed to fetch docker daemon info: %v","messagePattern":"failed to fetch docker daemon info: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"drivers/docker/driver.go","lineNumber":260,"sourceCode":"\nfunc (d *Driver) RecoverTask(handle *drivers.TaskHandle) error {\n\tif _, ok := d.tasks.Get(handle.Config.ID); ok {\n\t\treturn nil\n\t}\n\n\tvar handleState taskHandleState\n\tif err := handle.GetDriverState(&handleState); err != nil {\n\t\treturn fmt.Errorf(\"failed to decode driver task state: %v\", err)\n\t}\n\n\tdockerClient, err := d.getDockerClient()\n\tif err != nil {\n\t\treturn fmt.Errorf(\"failed to get docker client: %w\", err)\n\t}\n\n\tdockerInfo, err := dockerClient.Info(d.ctx, mclient.InfoOptions{})\n\tif err != nil {\n\t\treturn fmt.Errorf(\"failed to fetch docker daemon info: %v\", err)\n\t}\n\n\tinfinityClient, err := d.getInfinityClient()\n\tif err != nil {\n\t\treturn fmt.Errorf(\"failed to get docker long operations client: %w\", err)\n\t}\n\n\tcontainer, err := dockerClient.ContainerInspect(d.ctx, handleState.ContainerID, mclient.ContainerInspectOptions{})\n\tif err != nil {\n\t\treturn fmt.Errorf(\"failed to inspect container for id %q: %v\", handleState.ContainerID, err)\n\t}\n\n\th := &taskHandle{\n\t\tdockerClient:            dockerClient,\n\t\tdockerCGroupDriver:      dockerInfo.Info.CgroupDriver,\n\t\tinfinityClient:          infinityClient,\n\t\tlogger:                  d.logger.With(\"container_id\", container.Container.ID),\n\t\ttask:                    handle.Config,","sourceCodeStart":242,"sourceCodeEnd":278,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/drivers/docker/driver.go#L242-L278","documentation":"RecoverTask rebuilds an in-memory taskHandle for a task that survived a client/agent restart. Before doing so it queries the Docker daemon (dockerClient.Info) to learn the daemon's cgroup driver. If the daemon API call fails — daemon unreachable, socket not accessible, TLS/auth misconfigured, or daemon restarting — the recovery is aborted with this wrapped error.","triggerScenarios":"Docker daemon not running; DOCKER_HOST pointing at a wrong/unreachable host; Unix socket permissions denied; TLS certificates invalid or expired; daemon restarting during a client crash-recovery window; API version mismatch causing the Info request to fail.","commonSituations":"Nominator/agent host rebooted and dockerd comes up slower than the driver recovers tasks; running the client as a non-root user without docker group membership; DOCKER_HOST changed between restarts; firewall blocking the TCP docker endpoint on remote hosts.","solutions":["Verify dockerd is running and reachable: run `docker info` with the same DOCKER_HOST/environment the driver uses.","Check socket permissions / group membership (`usermod -aG docker <user>`) or the TCP/TLS endpoint config in the driver's docker endpoint setting.","If TLS is configured, validate docker.tls cert/key/CA paths and expiry; test with `docker --tlsverify info`.","Add retry/backoff around task recovery or ordering so recovery runs after dockerd is healthy."],"exampleFix":"// before\ndockerInfo, err := dockerClient.Info(d.ctx, mclient.InfoOptions{})\nif err != nil {\n\treturn fmt.Errorf(\"failed to fetch docker daemon info: %v\", err)\n}\n// after\ndockerInfo, err := dockerClient.Info(d.ctx, mclient.InfoOptions{})\nif err != nil {\n\tif isRetryable(err) && d.recoverRetry(ctx) == nil { // wait/retry while daemon comes up\n\t\tdockerInfo, err = dockerClient.Info(d.ctx, mclient.InfoOptions{})\n\t}\n\tif err != nil {\n\t\treturn fmt.Errorf(\"failed to fetch docker daemon info: %w\", err)\n\t}\n}","handlingStrategy":"retry","validationCode":"// before relying on recovery, confirm the daemon is reachable\n// (same env/endpoint the driver uses)\nfunc dockerDaemonReachable(endpoint string) error {\n\tcli, err := client.NewClientWithOpts(client.WithHost(endpoint), client.WithAPIVersionNegotiation())\n\tif err != nil {\n\t\treturn err\n\t}\n\tdefer cli.Close()\n\t_, err = cli.Info(context.Background())\n\treturn err\n}","typeGuard":null,"tryCatchPattern":"// Go: unwrap and retry transient daemon-connect errors\nerr := driver.RecoverTask(handle)\nif err != nil && strings.Contains(err.Error(), \"failed to fetch docker daemon info\") {\n\t// exponential backoff while dockerd comes up\n\tfor i := 0; i < 5; i++ {\n\t\ttime.Sleep(time.Duration(1<<i) * time.Second)\n\t\tif err = driver.RecoverTask(handle); err == nil {\n\t\t\tbreak\n\t\t}\n\t}\n}","preventionTips":["Ensure dockerd is enabled and started before the agent (systemd After=docker.service).","Pin DOCKER_HOST and keep it identical across restarts.","Grant the agent user docker group / socket access and test with `docker info`.","Monitor daemon health and TLS cert expiry on hosts running docker tasks."],"tags":["docker","daemon-connectivity","task-recovery","network"],"backgroundTag":"docker-daemon-unreachable","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}