What causes "error creating fifo %v: %w" in hashicorp/nomad?

Remove the stale file at the path and retry (EEXIST); restart the Nomad client so the alloc dir is rebuilt.

error creating fifo %v: %w

Error message

error creating fifo %v: %w

What it means

fifo.CreateAndRead creates a named pipe (FIFO) at the given path via mkfifo and returns a reader-opening function. This error wraps a mkfifo failure, so the log-monitoring FIFO for a task's stdout/stderr could not be created and log streaming cannot be set up.

Source

Thrown at client/lib/fifo/fifo_unix.go:24

package fifo

import (
	"fmt"
	"io"
	"os"
	"path/filepath"
)

// CreateAndRead creates a fifo at the given path, and returns an open function
// for reading. For compatibility with windows, the fifo must not exist
// already.
//
// It returns a reader open function that may block until a writer opens
// so it's advised to run it in a goroutine different from reader goroutine
func CreateAndRead(path string) (func() (io.ReadCloser, error), error) {
	// create first
	if err := mkfifo(path, 0600); err != nil {
		return nil, fmt.Errorf("error creating fifo %v: %w", path, err)
	}

	return func() (io.ReadCloser, error) {
		return OpenReader(path)
	}, nil
}

func OpenReader(path string) (io.ReadCloser, error) {
	dir := filepath.Dir(path)
	base := filepath.Base(path)

	root, err := os.OpenRoot(dir)
	if err != nil {
		return nil, fmt.Errorf("error opening fifo parent directory %q: %w", dir, err)
	}
	defer root.Close()

	// also uses O_NOFOLLOW under the hood

View on GitHub (pinned to 482b49bf1a)

Solutions

Remove the stale file at the path and retry (EEXIST); restart the Nomad client so the alloc dir is rebuilt.
Ensure the parent alloc directory exists and is writable by the Nomad agent user.
If the filesystem does not support FIFOs, move the data/alloc directories to a local filesystem (e.g. ext4/xfs) via client data_dir config.
Shorten the path if ENAMETOOLONG (adjust data_dir depth).

Example fix

// before: stale fifo causes EEXIST
$ ls /var/lib/nomad/alloc/.../alloc/logs/task.stdout.fifo
// after: clean stale fifo and restart
$ sudo rm /var/lib/nomad/alloc/.../alloc/logs/task.stdout.fifo
$ sudo systemctl restart nomad

Defensive patterns

Strategy: try-catch

Validate before calling

func fifoCreatable(path string) error {
  dir := filepath.Dir(path)
  if st, err := os.Stat(dir); err != nil || !st.IsDir() { return fmt.Errorf("dir missing: %s", dir) }
  if err := syscall.Access(dir, os.W_OK); err != nil { return err }
  if _, err := os.Stat(path); err == nil { return fmt.Errorf("stale file exists: %s", path) }
  return nil
}

Type guard

func fifoSupported(path string) bool {
  return syscall.Mkfifo(path, 0600) == nil || syscall.Errno(syscall.EEXIST) != syscall.Errno(0)
}

Try / catch

openFn, err := fifo.CreateAndRead(path)
if err != nil {
  var se syscall.Errno
  if errors.As(err, &se) && (errors.Is(err, syscall.EEXIST) || errors.Is(err, syscall.ENOENT)) {
    os.Remove(path) // clear stale fifo
    os.MkdirAll(filepath.Dir(path), 0755)
    openFn, err = fifo.CreateAndRead(path)
  }
  if err != nil { return err }
}

Prevention

Clean stale FIFOs after unclean client shutdowns (handle EEXIST).
Place data_dir/alloc dirs on a local filesystem that supports FIFOs (ext4/xfs), not restrictive NFS.
Ensure alloc directories exist and are writable by the agent user.
Keep alloc path depth short to avoid ENAMETOOLONG.

When it happens

Trigger: mkfifo(path, 0600) returns EEXIST (file already exists at path from a prior run), EACCES (cannot create in the directory), ENOENT (parent directory missing), or ENAMETOOLONG.

Common situations: Stale FIFO left behind after an unclean Nomad client shutdown; alloc directory permissions changed or partially deleted; task working directory on a filesystem that does not support FIFOs (some NFS/overlay configurations); path too long from deep alloc IDs.

Related errors

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/0a514fa4b97b353b. Report an issue: GitHub.