hashicorp/nomad · error

failed to fetch docker daemon info: %v

Error message

failed to fetch docker daemon info: %v

What it means

RecoverTask rebuilds an in-memory taskHandle for a task that survived a client/agent restart. Before doing so it queries the Docker daemon (dockerClient.Info) to learn the daemon's cgroup driver. If the daemon API call fails — daemon unreachable, socket not accessible, TLS/auth misconfigured, or daemon restarting — the recovery is aborted with this wrapped error.

Source

Thrown at drivers/docker/driver.go:260

func (d *Driver) RecoverTask(handle *drivers.TaskHandle) error {
	if _, ok := d.tasks.Get(handle.Config.ID); ok {
		return nil
	}

	var handleState taskHandleState
	if err := handle.GetDriverState(&handleState); err != nil {
		return fmt.Errorf("failed to decode driver task state: %v", err)
	}

	dockerClient, err := d.getDockerClient()
	if err != nil {
		return fmt.Errorf("failed to get docker client: %w", err)
	}

	dockerInfo, err := dockerClient.Info(d.ctx, mclient.InfoOptions{})
	if err != nil {
		return fmt.Errorf("failed to fetch docker daemon info: %v", err)
	}

	infinityClient, err := d.getInfinityClient()
	if err != nil {
		return fmt.Errorf("failed to get docker long operations client: %w", err)
	}

	container, err := dockerClient.ContainerInspect(d.ctx, handleState.ContainerID, mclient.ContainerInspectOptions{})
	if err != nil {
		return fmt.Errorf("failed to inspect container for id %q: %v", handleState.ContainerID, err)
	}

	h := &taskHandle{
		dockerClient:            dockerClient,
		dockerCGroupDriver:      dockerInfo.Info.CgroupDriver,
		infinityClient:          infinityClient,
		logger:                  d.logger.With("container_id", container.Container.ID),
		task:                    handle.Config,

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Verify dockerd is running and reachable: run `docker info` with the same DOCKER_HOST/environment the driver uses.
2. Check socket permissions / group membership (`usermod -aG docker <user>`) or the TCP/TLS endpoint config in the driver's docker endpoint setting.
3. If TLS is configured, validate docker.tls cert/key/CA paths and expiry; test with `docker --tlsverify info`.
4. Add retry/backoff around task recovery or ordering so recovery runs after dockerd is healthy.

Example fix

// before
dockerInfo, err := dockerClient.Info(d.ctx, mclient.InfoOptions{})
if err != nil {
	return fmt.Errorf("failed to fetch docker daemon info: %v", err)
}
// after
dockerInfo, err := dockerClient.Info(d.ctx, mclient.InfoOptions{})
if err != nil {
	if isRetryable(err) && d.recoverRetry(ctx) == nil { // wait/retry while daemon comes up
		dockerInfo, err = dockerClient.Info(d.ctx, mclient.InfoOptions{})
	}
	if err != nil {
		return fmt.Errorf("failed to fetch docker daemon info: %w", err)
	}
}

// before relying on recovery, confirm the daemon is reachable
// (same env/endpoint the driver uses)
func dockerDaemonReachable(endpoint string) error {
	cli, err := client.NewClientWithOpts(client.WithHost(endpoint), client.WithAPIVersionNegotiation())
	if err != nil {
		return err
	}
	defer cli.Close()
	_, err = cli.Info(context.Background())
	return err
}

// Go: unwrap and retry transient daemon-connect errors
err := driver.RecoverTask(handle)
if err != nil && strings.Contains(err.Error(), "failed to fetch docker daemon info") {
	// exponential backoff while dockerd comes up
	for i := 0; i < 5; i++ {
		time.Sleep(time.Duration(1<<i) * time.Second)
		if err = driver.RecoverTask(handle); err == nil {
			break
		}
	}
}

When it happens

Trigger: Docker daemon not running; DOCKER_HOST pointing at a wrong/unreachable host; Unix socket permissions denied; TLS certificates invalid or expired; daemon restarting during a client crash-recovery window; API version mismatch causing the Info request to fail.

Common situations: Nominator/agent host rebooted and dockerd comes up slower than the driver recovers tasks; running the client as a non-root user without docker group membership; DOCKER_HOST changed between restarts; firewall blocking the TCP docker endpoint on remote hosts.

Related errors

• failed to get docker long operations client: %w
• failed to inspect container for id %q: %v
• failed to setup replacement docker logger: %v
• failed to store driver state: %v
• Port %q not found, check network block

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/3818d1f817451a5c. Report an issue: GitHub.