hashicorp/nomad · error

failed to create fifo for extracting logs: %v

Error message

failed to create fifo for extracting logs: %v

What it means

newLogRotatorWrapper sets up the FIFO (named pipe) that task processes write logs into so the rotator can consume them; when the FIFO cannot be stat'd, created, or opened, this error is returned after logging the stat and create errors. Without the FIFO, the task's stdout/stderr cannot be captured.

Source

Thrown at client/logmon/logmon.go:213

// processOutWriter to attach to the stdout or stderr of a process.
func newLogRotatorWrapper(path string, logger hclog.Logger, rotator io.WriteCloser) (*logRotatorWrapper, error) {
	logger.Debug("opening fifo", "path", path)

	var openFn func() (io.ReadCloser, error)
	var err error

	_, serr := os.Stat(path)
	if os.IsNotExist(serr) {
		openFn, err = fifo.CreateAndRead(path)
	} else {
		openFn = func() (io.ReadCloser, error) {
			return fifo.OpenReader(path)
		}
	}

	if err != nil {
		logger.Error("failed to create FIFO", "stat_error", serr, "create_err", err)
		return nil, fmt.Errorf("failed to create fifo for extracting logs: %v", err)
	}

	wrap := &logRotatorWrapper{
		fifoPath:          path,
		rotatorWriter:     rotator,
		hasFinishedCopied: make(chan struct{}),
		openCompleted:     make(chan struct{}),
		logger:            logger,
	}

	wrap.start(openFn)
	return wrap, nil
}

// start starts a goroutine that copies from the pipe into the rotator. This is
// called by the constructor and not the user of the wrapper.
func (l *logRotatorWrapper) start(openFn func() (io.ReadCloser, error)) {
	go func() {

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Inspect the logged stat_error and create_err fields for the underlying cause
2. Remove the stale/incompatible file at the FIFO path so mkfifo can recreate it
3. Ensure the log directory is writable and on a filesystem supporting named pipes (avoid exotic NFS configs)
4. Restart the client/logmon after cleaning the log directory so the FIFO is recreated
5. Check process privileges and SELinux/AppAllow policies for mkfifo in that path

Example fix

// before
wrapper, err := newLogRotatorWrapper("/logs/task.fifo", ...) // stale file blocks mkfifo
// after
if fi, err := os.Stat("/logs/task.fifo"); err == nil && fi.Mode()&os.ModeNamedPipe == 0 {
    os.Remove("/logs/task.fifo") // clear incompatible leftover
}
wrapper, err := newLogRotatorWrapper("/logs/task.fifo", ...)

func ensureFifoCreatable(path string) error {
    dir := filepath.Dir(path)
    if err := os.MkdirAll(dir, 0o755); err != nil {
        return fmt.Errorf("fifo dir not creatable: %w", err)
    }
    if fi, err := os.Stat(path); err == nil && fi.Mode()&os.ModeNamedPipe == 0 {
        if err := os.Remove(path); err != nil {
            return fmt.Errorf("stale non-fifo file blocks mkfifo: %w", err)
        }
    }
    return nil
}

wrapper, err := newLogRotatorWrapper(fifoPath, logger, rotator)
if err != nil {
    if strings.Contains(err.Error(), "failed to create fifo") {
        _ = os.Remove(fifoPath) // clear stale path and retry once
        wrapper, err = newLogRotatorWrapper(fifoPath, logger, rotator)
    }
    if err != nil {
        return nil, err
    }
}

When it happens

Trigger: FIFO path invalid or in an unwritable directory (mkfifo fails), an incompatible non-FIFO file already exists at the path, os.Stat failing unexpectedly, or fifo.Open failing to open the created pipe.

Common situations: Stale FIFO file left by a previous crashed client with wrong permissions; log directory on a filesystem without FIFO support (some network/NFS mounts); resource exhaustion; client restarted into a dirty log dir.

Related errors

• error creating fifo %v: %w
• error opening fifo parent directory %q: %w
• failed to create stdout logfile for %q: %v
• failed to create stderr logfile for %q: %v
• error opening reader at %s: %w

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/6f371c0001f6c6ea. Report an issue: GitHub.