hashicorp/nomad · error

failed to attach to exec object: %w

Error message

failed to attach to exec object: %w

What it means

After creating the exec object, the driver hijacks its input/output streams with ExecAttach. If attaching the stream to the exec instance fails at the Docker API level, this wrapped error is returned. The exec may exist but its streams could not be connected.

Source

Thrown at drivers/docker/handle.go:108

		AttachStdout: true,
		AttachStderr: true,
		TTY:          false,
		Cmd:          fullCmd,
	}
	exec, err := h.dockerClient.ExecCreate(ctx, h.containerID, createExecOpts)
	if err != nil {
		return nil, fmt.Errorf("failed to create exec object: %v", err)
	}

	execResult := &drivers.ExecTaskResult{ExitResult: &drivers.ExitResult{}}
	stdout, _ := circbuf.NewBuffer(int64(drivers.CheckBufSize))
	stderr, _ := circbuf.NewBuffer(int64(drivers.CheckBufSize))
	startOpts := mclient.ExecAttachOptions{TTY: false}

	// hijack exec output streams
	hijacked, err := h.dockerClient.ExecAttach(ctx, exec.ID, startOpts)
	if err != nil {
		return nil, fmt.Errorf("failed to attach to exec object: %w", err)
	}

	_, err = stdcopy.StdCopy(stdout, stderr, hijacked.Reader)
	if err != nil {
		return nil, err
	}
	defer hijacked.Close()

	execResult.Stdout = stdout.Bytes()
	execResult.Stderr = stderr.Bytes()
	res, err := h.dockerClient.ExecInspect(ctx, exec.ID, mclient.ExecInspectOptions{})
	if err != nil {
		return execResult, fmt.Errorf("failed to inspect exit code of exec object: %w", err)
	}

	execResult.ExitResult.ExitCode = res.ExitCode
	return execResult, nil
}

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Retry the exec command; transient attach failures usually succeed on a second attempt.
2. If using a TCP Docker endpoint, ensure the proxy/LB supports HTTP upgrade and long-lived hijacked connections.
3. Check Docker daemon logs for errors around the exec attach and restart the daemon if degraded.
4. Verify the container is still running so the exec ID remains attachable.

Example fix

// before: attach over flaky TCP endpoint
hosts = ["tcp://docker-host:2375"]
// after: use local unix socket for reliable hijack
hosts = ["unix:///var/run/docker.sock"]

// ensure a plain, upgrade-capable connection to the docker daemon
// prefer unix:///var/run/docker.sock over proxied TCP endpoints

res, err := driver.ExecTask(ctx, taskID, opts)
if err != nil && strings.Contains(err.Error(), "failed to attach to exec object") {
    // transient hijack failure: retry once after backoff
    time.Sleep(time.Second)
    res, err = driver.ExecTask(ctx, taskID, opts)
}

When it happens

Trigger: h.dockerClient.ExecAttach(ctx, exec.ID, startOpts) errors: exec instance was removed before attach, Docker daemon connection dropped mid-handshake, engine returned HTTP upgrade/connection error on the hijacked endpoint.

Common situations: Docker daemon restarted between create and attach; network/proxy between Nomad and a remote Docker endpoint drops hijacked connections (common with TCP Docker endpoints behind load balancers that don't support connection upgrade); race with container removal.

Related errors

• failed to create exec object: %v
• failed to inspect exit code of exec object: %w
• command is not present
• not a terminal
• no exec command is configured

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/687f4e66f33a97af. Report an issue: GitHub.