{"record":{"id":"687f4e66f33a97af","repo":"hashicorp/nomad","slug":"failed-to-attach-to-exec-object-w","errorCode":null,"errorMessage":"failed to attach to exec object: %w","messagePattern":"failed to attach to exec object: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"drivers/docker/handle.go","lineNumber":108,"sourceCode":"\t\tAttachStdout: true,\n\t\tAttachStderr: true,\n\t\tTTY:          false,\n\t\tCmd:          fullCmd,\n\t}\n\texec, err := h.dockerClient.ExecCreate(ctx, h.containerID, createExecOpts)\n\tif err != nil {\n\t\treturn nil, fmt.Errorf(\"failed to create exec object: %v\", err)\n\t}\n\n\texecResult := &drivers.ExecTaskResult{ExitResult: &drivers.ExitResult{}}\n\tstdout, _ := circbuf.NewBuffer(int64(drivers.CheckBufSize))\n\tstderr, _ := circbuf.NewBuffer(int64(drivers.CheckBufSize))\n\tstartOpts := mclient.ExecAttachOptions{TTY: false}\n\n\t// hijack exec output streams\n\thijacked, err := h.dockerClient.ExecAttach(ctx, exec.ID, startOpts)\n\tif err != nil {\n\t\treturn nil, fmt.Errorf(\"failed to attach to exec object: %w\", err)\n\t}\n\n\t_, err = stdcopy.StdCopy(stdout, stderr, hijacked.Reader)\n\tif err != nil {\n\t\treturn nil, err\n\t}\n\tdefer hijacked.Close()\n\n\texecResult.Stdout = stdout.Bytes()\n\texecResult.Stderr = stderr.Bytes()\n\tres, err := h.dockerClient.ExecInspect(ctx, exec.ID, mclient.ExecInspectOptions{})\n\tif err != nil {\n\t\treturn execResult, fmt.Errorf(\"failed to inspect exit code of exec object: %w\", err)\n\t}\n\n\texecResult.ExitResult.ExitCode = res.ExitCode\n\treturn execResult, nil\n}","sourceCodeStart":90,"sourceCodeEnd":126,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/drivers/docker/handle.go#L90-L126","documentation":"After creating the exec object, the driver hijacks its input/output streams with ExecAttach. If attaching the stream to the exec instance fails at the Docker API level, this wrapped error is returned. The exec may exist but its streams could not be connected.","triggerScenarios":"h.dockerClient.ExecAttach(ctx, exec.ID, startOpts) errors: exec instance was removed before attach, Docker daemon connection dropped mid-handshake, engine returned HTTP upgrade/connection error on the hijacked endpoint.","commonSituations":"Docker daemon restarted between create and attach; network/proxy between Nomad and a remote Docker endpoint drops hijacked connections (common with TCP Docker endpoints behind load balancers that don't support connection upgrade); race with container removal.","solutions":["Retry the exec command; transient attach failures usually succeed on a second attempt.","If using a TCP Docker endpoint, ensure the proxy/LB supports HTTP upgrade and long-lived hijacked connections.","Check Docker daemon logs for errors around the exec attach and restart the daemon if degraded.","Verify the container is still running so the exec ID remains attachable."],"exampleFix":"// before: attach over flaky TCP endpoint\nhosts = [\"tcp://docker-host:2375\"]\n// after: use local unix socket for reliable hijack\nhosts = [\"unix:///var/run/docker.sock\"]","handlingStrategy":"retry","validationCode":"// ensure a plain, upgrade-capable connection to the docker daemon\n// prefer unix:///var/run/docker.sock over proxied TCP endpoints","typeGuard":null,"tryCatchPattern":"res, err := driver.ExecTask(ctx, taskID, opts)\nif err != nil && strings.Contains(err.Error(), \"failed to attach to exec object\") {\n    // transient hijack failure: retry once after backoff\n    time.Sleep(time.Second)\n    res, err = driver.ExecTask(ctx, taskID, opts)\n}","preventionTips":["Avoid LBs/proxies in front of Docker TCP endpoints that break hijacked streams.","Watch for daemon restarts (docker events) around exec usage.","Retry exec operations; attach failures are frequently transient."],"tags":["docker-driver","exec","stream-attach"],"backgroundTag":"docker-exec-attach-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}