What causes "newWorker: failed to create stdin pipe" in ory/hydra?

Check fd usage (`lsof -p | wc -l`) and raise the limit (`ulimit -n`, systemd LimitNOFILE, container ulimits)

newWorker: failed to create stdin pipe

Error message

newWorker: failed to create stdin pipe

What it means

`newWorker` builds the child `exec.Cmd` for the jsonnet worker process; `cmd.StdinPipe()` failed, so no stdin pipe could be created and the worker construction aborts. In practice this almost always means the process is out of file descriptors, since pipe creation requires two fds.

Solutions

Check fd usage (`lsof -p <pid> | wc -l`) and raise the limit (`ulimit -n`, systemd LimitNOFILE, container ulimits)
Reduce the jsonnet worker pool size — NewProcessPool enforces a minimum of 5, so very small containers may need a bigger fd budget
Look for fd leaks: worker processes that are never destroyed keep pipes open; ensure the pool is Closed on shutdown
Restart the process to clear leaked fds if the pool has been running a long time

Example fix

// before: large pool on a box with low ulimit
pool := jsonnetsecure.NewProcessPool(100) // needs ~200+ fds
// after: size the pool to the environment
pool := jsonnetsecure.NewProcessPool(runtime.NumCPU()) // and raise ulimit -n accordingly

Defensive patterns

Strategy: validation

Validate before calling

// check fd headroom before creating the pool
fds, _ := filepath.Glob("/proc/self/fd/*")
var lim syscall.Rlimit
syscall.Getrlimit(syscall.RLIMIT_NOFILE, &lim)
if len(fds) > int(lim.Cur)-2*poolSize {
    return errors.New("not enough file descriptors for worker pool")
}

Prevention

Set a generous RLIMIT_NOFILE (e.g. 65536) for the service and containers
Call Pool.Close() on shutdown so worker pipes are released
Right-size the pool (each worker consumes several fds) instead of maxing it out
Alert on open-fd count approaching the limit in production

When it happens

Trigger: Called by the puddle pool constructor when spinning up a worker: `cmd.StdinPipe()` returns an error — overwhelmingly `EMFILE`/`ENFILE` (fd limit reached) from too many open files/sockets in the parent process.

Common situations: Large worker pool sizes combined with many open connections/files exhausting the fd limit; leaking worker processes/pipes over time so each new worker needs fds that no longer exist; low `ulimit -n` in containers.

Related errors

AI-assisted analysis of ory/hydra@4174065ffb (2026-09-03). Data as JSON: /api/errors/903e84d253387a0f. Report an issue: GitHub.

Appendix: source

Thrown at oryx/jsonnetsecure/jsonnet_pool.go:136

func newWorker(ctx context.Context) (_ worker, err error) {
	tracer := trace.SpanFromContext(ctx).TracerProvider().Tracer("")
	ctx, span := tracer.Start(ctx, "jsonnetsecure.newWorker")
	defer otelx.End(span, &err)

	path, _ := ctx.Value(contextValuePath).(string)
	if path == "" {
		return worker{}, errors.New("newWorker: missing binary path in context")
	}
	args, _ := ctx.Value(contextValueArgs).([]string)
	cmd := exec.Command(path, append(args, "-0")...)
	cmd.Env = []string{"GOMAXPROCS=1"}
	cmd.WaitDelay = 100 * time.Millisecond

	span.SetAttributes(semconv.ProcessCommand(cmd.Path), semconv.ProcessCommandArgs(cmd.Args...))

	stdin, err := cmd.StdinPipe()
	if err != nil {
		return worker{}, errors.Wrap(err, "newWorker: failed to create stdin pipe")
	}

	in := make(chan []byte, 1)
	go func(c <-chan []byte) {
		for input := range c {
			if _, err := stdin.Write(append(input, 0)); err != nil {
				stdin.Close()
				return
			}
		}
	}(in)

	stdout, err := cmd.StdoutPipe()
	if err != nil {
		return worker{}, errors.Wrap(err, "newWorker: failed to create stdout pipe")
	}
	stderr, err := cmd.StderrPipe()
	if err != nil {

View on GitHub (pinned to 4174065ffb)