ory/hydra · critical

newWorker: failed to start process

Error message

newWorker: failed to start process

What it means

`cmd.Start()` failed when launching the jsonnet worker subprocess, so `newWorker` returns this wrapped error. The pipe creation succeeded but the OS could not fork/exec the binary — most commonly the binary path doesn't exist or isn't executable, or resource limits (fork/memory) blocked it.

Source

Thrown at oryx/jsonnetsecure/jsonnet_pool.go:159

		for input := range c {
			if _, err := stdin.Write(append(input, 0)); err != nil {
				stdin.Close()
				return
			}
		}
	}(in)

	stdout, err := cmd.StdoutPipe()
	if err != nil {
		return worker{}, errors.Wrap(err, "newWorker: failed to create stdout pipe")
	}
	stderr, err := cmd.StderrPipe()
	if err != nil {
		return worker{}, errors.Wrap(err, "newWorker: failed to create stderr pipe")
	}

	if err := cmd.Start(); err != nil {
		return worker{}, errors.Wrap(err, "newWorker: failed to start process")
	}

	span.SetAttributes(semconv.ProcessPID(cmd.Process.Pid))

	scan := func(c chan<- string, r io.Reader, maxTokenSize int) {
		defer close(c)
		scanner := bufio.NewScanner(r)
		scanner.Buffer(make([]byte, 0, 64*KiB), maxTokenSize)

		scanner.Split(splitNull)
		for scanner.Scan() {
			c <- scanner.Text()
		}
		if err := scanner.Err(); err != nil {
			c <- "ERROR: scan: " + err.Error()
		}
	}
	out := make(chan string, 1)

View on GitHub (pinned to 4174065ffb)

Solutions

1. Verify the configured binary path exists and is executable (`ls -l <path>`); fix opts.jsonnetBinaryPath
2. Check the wrapped error via `errors.Cause` — `no such file` means wrong path, `permission denied` means chmod/ACL, `resource temporarily unavailable` means fork limits
3. Ensure the binary is present in the deployment image and matches the container architecture
4. Check cgroup limits (pids.max, memory) and ulimits if exec fails under load

Example fix

// before: path assumed, never checked
vm := jsonnetsecure.NewProcessPoolVM(&jsonnetsecure.VMOptions{JsonnetBinaryPath: "/usr/local/bin/jsonnet"})
// after: verify the binary before wiring the pool
if _, err := os.Stat(binPath); err != nil {
    log.Fatalf("jsonnet binary missing: %v", err)
}
if err := syscall.Access(binPath, syscall.X_OK); err != nil {
    log.Fatalf("jsonnet binary not executable: %v", err)
}
vm := jsonnetsecure.NewProcessPoolVM(&jsonnetsecure.VMOptions{JsonnetBinaryPath: binPath})

// validate the binary before constructing the pool VM
func validateJsonnetBinary(path string) error {
    fi, err := os.Stat(path)
    if err != nil { return fmt.Errorf("jsonnet binary missing: %w", err) }
    if fi.IsDir() { return errors.New("path is a directory") }
    if fi.Mode()&0o111 == 0 { return errors.New("jsonnet binary is not executable") }
    return nil
}

worker, err := vm.pool.puddle.Acquire(ctx)
if err != nil {
    var execErr *exec.Error
    if errors.As(errors.Cause(err), &execErr) {
        log.Fatalf("jsonnet binary %q unusable: %v", execErr.Name, execErr.Err)
    }
    return "", errors.Wrap(err, "jsonnetsecure: acquire")
}

When it happens

Trigger: Pool construction or refill calls `newWorker`, which calls `cmd.Start()` on the configured jsonnet binary path (from contextValuePath); exec fails with e.g. `no such file or directory`, `permission denied`, or EAGAIN under fork limits (cgroup PIDs/memory limits).

Common situations: Wrong `jsonnetBinaryPath` in vmOptions after a packaging change; the binary wasn't copied into a slim container image; missing execute bit; cgroup pids.max or memory limits preventing fork; binary built for the wrong architecture.

Related errors

• failed to write json output
• newWorker: failed to create stdin pipe
• newWorker: failed to create stdout pipe
• newWorker: failed to create stderr pipe
• newWorker: warm up failed

AI-assisted analysis of ory/hydra@4174065ffb (2026-09-03). Data as JSON: /api/errors/81d7e88a1f866724. Report an issue: GitHub.