hashicorp/nomad · error

failed to set command io: %v

Error message

failed to set command io: %v

What it means

runNoTTY wraps a failure from e.setIO(), which wires the process's stdin/stdout/stderr to io.Pipes. This means the executor could not attach the configured I/O streams to the child command before starting it.

Source

Thrown at drivers/shared/executor/exec_utils.go:112

func (e *execHelper) runNoTTY(ctx context.Context, stream drivers.ExecTaskStream) error {
	var sendLock sync.Mutex
	send := func(v *drivers.ExecTaskStreamingResponseMsg) error {
		sendLock.Lock()
		defer sendLock.Unlock()

		return stream.Send(v)
	}

	stdinPr, stdinPw := io.Pipe()
	stdoutPr, stdoutPw := io.Pipe()
	stderrPr, stderrPw := io.Pipe()

	defer stdoutPw.Close()
	defer stderrPw.Close()

	if err := e.setIO(stdinPr, stdoutPw, stderrPw); err != nil {
		return fmt.Errorf("failed to set command io: %v", err)
	}

	if err := e.processStart(); err != nil {
		return fmt.Errorf("failed to start command: %v", err)
	}

	var wg sync.WaitGroup
	errCh := make(chan error, 3)

	wg.Add(2)
	go handleStdin(e.logger, stdinPw, stream, errCh)
	go handleStdout(e.logger, stdoutPr, &wg, send, errCh)
	go handleStderr(e.logger, stderrPr, &wg, send, errCh)

	ps, err := e.processWait()

	// force close streams to close out the stream copying goroutines
	stdinPr.Close()

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Read the wrapped %v cause — fd exhaustion ('too many open files') is the most common; raise ulimit -n
2. Verify the executor's setIO implementation correctly assigns cmd.Stdin/Stdout/Stderr
3. Check for leaked file descriptors in the nomad client process (ls /proc/<pid>/fd)
4. Retry the task; if persistent, update the nomad/drivers package version

Example fix

// before
ulimit -n 256
// after
ulimit -n 65536  # and set LimitNOFILE in the systemd unit

func fdsAvailable(threshold uint64) bool {
  var r syscall.Rlimit
  if err := syscall.Getrlimit(syscall.RLIMIT_NOFILE, &r); err != nil { return false }
  ents, _ := os.ReadDir("/proc/self/fd")
  return uint64(len(ents)) < r.Cur-threshold
}

if err := launchNoTTY(); err != nil {
  if strings.Contains(err.Error(), "failed to set command io") {
    if fdsAvailable(64) { return retryLaunch() }
    return fmt.Errorf("fd exhaustion suspected: %w", err)
  }
  return err
}

When it happens

Trigger: run() with Tty=false reaches `e.setIO(stdinPr, stdoutPw, stderrPw)` and it returns an error from configuring cmd.Stdin/Stdout/Stderr on the underlying exec.Cmd.

Common situations: Custom executor implementations whose setIO misbehaves, OS failures assigning pipe file descriptors (fd exhaustion), rare driver bugs after upgrade.

Related errors

• failed to get pty: %v
• unable to read rooted allocation directory
• can't seek to offset %d: %w
• Couldn't copy %q to %q: %w
• failed to encode bootstrap command line: %w

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/03b3bd376f18b985. Report an issue: GitHub.