hashicorp/nomad · error

failed to get docker long operations client: %w

Error message

failed to get docker long operations client: %w

What it means

After fetching daemon info, RecoverTask obtains a secondary client (via d.getInfinityClient) used for long-running docker operations. Failure here means the driver could not construct that client from the driver's endpoint/TLS configuration, so recovery cannot proceed and the error is wrapped with %w so callers can unwrap the underlying cause.

Source

Thrown at drivers/docker/driver.go:265

	var handleState taskHandleState
	if err := handle.GetDriverState(&handleState); err != nil {
		return fmt.Errorf("failed to decode driver task state: %v", err)
	}

	dockerClient, err := d.getDockerClient()
	if err != nil {
		return fmt.Errorf("failed to get docker client: %w", err)
	}

	dockerInfo, err := dockerClient.Info(d.ctx, mclient.InfoOptions{})
	if err != nil {
		return fmt.Errorf("failed to fetch docker daemon info: %v", err)
	}

	infinityClient, err := d.getInfinityClient()
	if err != nil {
		return fmt.Errorf("failed to get docker long operations client: %w", err)
	}

	container, err := dockerClient.ContainerInspect(d.ctx, handleState.ContainerID, mclient.ContainerInspectOptions{})
	if err != nil {
		return fmt.Errorf("failed to inspect container for id %q: %v", handleState.ContainerID, err)
	}

	h := &taskHandle{
		dockerClient:            dockerClient,
		dockerCGroupDriver:      dockerInfo.Info.CgroupDriver,
		infinityClient:          infinityClient,
		logger:                  d.logger.With("container_id", container.Container.ID),
		task:                    handle.Config,
		containerID:             container.Container.ID,
		containerCgroup:         string(container.Container.HostConfig.Cgroup),
		containerImage:          container.Container.Image,
		doneCh:                  make(chan bool),
		waitCh:                  make(chan struct{}),

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Check the driver's docker endpoint and TLS config (cert/key/CA files exist and are readable) — the same settings that worked at task start.
2. Run `docker info` against the configured endpoint to confirm the daemon is dialable.
3. Restart the nomad client/plugin so the driver re-reads its config, then retry task recovery.
4. Inspect the unwrapped cause (`errors.Unwrap`) — the %w wrapping preserves the root error from the client factory.

Example fix

// before
infinityClient, err := d.getInfinityClient()
if err != nil {
	return fmt.Errorf("failed to get docker long operations client: %w", err)
}
// after
infinityClient, err := d.getInfinityClient()
if err != nil {
	d.logger.Error("infinity client setup failed", "endpoint", d.config.Endpoint, "error", err)
	return fmt.Errorf("failed to get docker long operations client: %w", err)
}

// validate driver docker config before starting/recovering tasks
func validateDockerDriverConfig(endpoint string, tls *TLSConfig) error {
	if endpoint == "" {
		return errors.New("docker endpoint is empty")
	}
	if tls != nil {
		for _, p := range []string{tls.Cert, tls.Key, tls.CA} {
			if p != "" {
				if _, err := os.Stat(p); err != nil {
					return fmt.Errorf("tls file missing: %s: %w", p, err)
				}
			}
		}
	}
	return nil
}

// Go: unwrap the %w cause to distinguish config vs connectivity
err := driver.RecoverTask(handle)
var root error
for e := err; e != nil; e = errors.Unwrap(e) {
	root = e
}
if err != nil && strings.Contains(err.Error(), "long operations client") {
	log.Printf("infinity client init failed, root cause: %v", root)
}

When it happens

Trigger: getInfinityClient fails because the docker endpoint is unset or malformed, the underlying client factory cannot dial the daemon, TLS cert/key/CA files are missing or unreadable, or client construction (API negotiation) returns an error.

Common situations: Driver plugin restarted with a stale or changed `docker endpoint` config; TLS secrets rotated or deleted between restarts; environment where DOCKER_HOST is only set in the original plugin process; memory/fd exhaustion preventing new client creation.

Related errors

• Failed to create docker client: %v
• failed to launch docker logger process %s: %v
• failed to get docker client: %w
• failed to fetch docker daemon info: %v
• failed to inspect container for id %q: %v

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/fca10a0cbef99021. Report an issue: GitHub.