{"record":{"id":"03b3bd376f18b985","repo":"hashicorp/nomad","slug":"failed-to-set-command-io-v","errorCode":null,"errorMessage":"failed to set command io: %v","messagePattern":"failed to set command io: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"drivers/shared/executor/exec_utils.go","lineNumber":112,"sourceCode":"\nfunc (e *execHelper) runNoTTY(ctx context.Context, stream drivers.ExecTaskStream) error {\n\tvar sendLock sync.Mutex\n\tsend := func(v *drivers.ExecTaskStreamingResponseMsg) error {\n\t\tsendLock.Lock()\n\t\tdefer sendLock.Unlock()\n\n\t\treturn stream.Send(v)\n\t}\n\n\tstdinPr, stdinPw := io.Pipe()\n\tstdoutPr, stdoutPw := io.Pipe()\n\tstderrPr, stderrPw := io.Pipe()\n\n\tdefer stdoutPw.Close()\n\tdefer stderrPw.Close()\n\n\tif err := e.setIO(stdinPr, stdoutPw, stderrPw); err != nil {\n\t\treturn fmt.Errorf(\"failed to set command io: %v\", err)\n\t}\n\n\tif err := e.processStart(); err != nil {\n\t\treturn fmt.Errorf(\"failed to start command: %v\", err)\n\t}\n\n\tvar wg sync.WaitGroup\n\terrCh := make(chan error, 3)\n\n\twg.Add(2)\n\tgo handleStdin(e.logger, stdinPw, stream, errCh)\n\tgo handleStdout(e.logger, stdoutPr, &wg, send, errCh)\n\tgo handleStderr(e.logger, stderrPr, &wg, send, errCh)\n\n\tps, err := e.processWait()\n\n\t// force close streams to close out the stream copying goroutines\n\tstdinPr.Close()","sourceCodeStart":94,"sourceCodeEnd":130,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/drivers/shared/executor/exec_utils.go#L94-L130","documentation":"runNoTTY wraps a failure from e.setIO(), which wires the process's stdin/stdout/stderr to io.Pipes. This means the executor could not attach the configured I/O streams to the child command before starting it.","triggerScenarios":"run() with Tty=false reaches `e.setIO(stdinPr, stdoutPw, stderrPw)` and it returns an error from configuring cmd.Stdin/Stdout/Stderr on the underlying exec.Cmd.","commonSituations":"Custom executor implementations whose setIO misbehaves, OS failures assigning pipe file descriptors (fd exhaustion), rare driver bugs after upgrade.","solutions":["Read the wrapped %v cause — fd exhaustion ('too many open files') is the most common; raise ulimit -n","Verify the executor's setIO implementation correctly assigns cmd.Stdin/Stdout/Stderr","Check for leaked file descriptors in the nomad client process (ls /proc/<pid>/fd)","Retry the task; if persistent, update the nomad/drivers package version"],"exampleFix":"// before\nulimit -n 256\n// after\nulimit -n 65536  # and set LimitNOFILE in the systemd unit","handlingStrategy":"validation","validationCode":"func fdsAvailable(threshold uint64) bool {\n  var r syscall.Rlimit\n  if err := syscall.Getrlimit(syscall.RLIMIT_NOFILE, &r); err != nil { return false }\n  ents, _ := os.ReadDir(\"/proc/self/fd\")\n  return uint64(len(ents)) < r.Cur-threshold\n}","typeGuard":null,"tryCatchPattern":"if err := launchNoTTY(); err != nil {\n  if strings.Contains(err.Error(), \"failed to set command io\") {\n    if fdsAvailable(64) { return retryLaunch() }\n    return fmt.Errorf(\"fd exhaustion suspected: %w\", err)\n  }\n  return err\n}","preventionTips":["Raise RLIMIT_NOFILE for the nomad client (LimitNOFILE=65536)","Audit the client for fd leaks under load","Keep executor packages up to date","Restart long-lived clients periodically"],"tags":["io","pipes","process","nomad-executor"],"backgroundTag":"io-pipe-setup-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}