{"record":{"id":"81d7e88a1f866724","repo":"ory/hydra","slug":"newworker-failed-to-start-process","errorCode":null,"errorMessage":"newWorker: failed to start process","messagePattern":"newWorker: failed to start process","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"oryx/jsonnetsecure/jsonnet_pool.go","lineNumber":159,"sourceCode":"\t\tfor input := range c {\n\t\t\tif _, err := stdin.Write(append(input, 0)); err != nil {\n\t\t\t\tstdin.Close()\n\t\t\t\treturn\n\t\t\t}\n\t\t}\n\t}(in)\n\n\tstdout, err := cmd.StdoutPipe()\n\tif err != nil {\n\t\treturn worker{}, errors.Wrap(err, \"newWorker: failed to create stdout pipe\")\n\t}\n\tstderr, err := cmd.StderrPipe()\n\tif err != nil {\n\t\treturn worker{}, errors.Wrap(err, \"newWorker: failed to create stderr pipe\")\n\t}\n\n\tif err := cmd.Start(); err != nil {\n\t\treturn worker{}, errors.Wrap(err, \"newWorker: failed to start process\")\n\t}\n\n\tspan.SetAttributes(semconv.ProcessPID(cmd.Process.Pid))\n\n\tscan := func(c chan<- string, r io.Reader, maxTokenSize int) {\n\t\tdefer close(c)\n\t\tscanner := bufio.NewScanner(r)\n\t\tscanner.Buffer(make([]byte, 0, 64*KiB), maxTokenSize)\n\n\t\tscanner.Split(splitNull)\n\t\tfor scanner.Scan() {\n\t\t\tc <- scanner.Text()\n\t\t}\n\t\tif err := scanner.Err(); err != nil {\n\t\t\tc <- \"ERROR: scan: \" + err.Error()\n\t\t}\n\t}\n\tout := make(chan string, 1)","sourceCodeStart":141,"sourceCodeEnd":177,"githubUrl":"https://github.com/ory/hydra/blob/4174065ffb052799890f7480f5360a877a67ffc1/oryx/jsonnetsecure/jsonnet_pool.go#L141-L177","documentation":"`cmd.Start()` failed when launching the jsonnet worker subprocess, so `newWorker` returns this wrapped error. The pipe creation succeeded but the OS could not fork/exec the binary — most commonly the binary path doesn't exist or isn't executable, or resource limits (fork/memory) blocked it.","triggerScenarios":"Pool construction or refill calls `newWorker`, which calls `cmd.Start()` on the configured jsonnet binary path (from contextValuePath); exec fails with e.g. `no such file or directory`, `permission denied`, or EAGAIN under fork limits (cgroup PIDs/memory limits).","commonSituations":"Wrong `jsonnetBinaryPath` in vmOptions after a packaging change; the binary wasn't copied into a slim container image; missing execute bit; cgroup pids.max or memory limits preventing fork; binary built for the wrong architecture.","solutions":["Verify the configured binary path exists and is executable (`ls -l <path>`); fix opts.jsonnetBinaryPath","Check the wrapped error via `errors.Cause` — `no such file` means wrong path, `permission denied` means chmod/ACL, `resource temporarily unavailable` means fork limits","Ensure the binary is present in the deployment image and matches the container architecture","Check cgroup limits (pids.max, memory) and ulimits if exec fails under load"],"exampleFix":"// before: path assumed, never checked\nvm := jsonnetsecure.NewProcessPoolVM(&jsonnetsecure.VMOptions{JsonnetBinaryPath: \"/usr/local/bin/jsonnet\"})\n// after: verify the binary before wiring the pool\nif _, err := os.Stat(binPath); err != nil {\n    log.Fatalf(\"jsonnet binary missing: %v\", err)\n}\nif err := syscall.Access(binPath, syscall.X_OK); err != nil {\n    log.Fatalf(\"jsonnet binary not executable: %v\", err)\n}\nvm := jsonnetsecure.NewProcessPoolVM(&jsonnetsecure.VMOptions{JsonnetBinaryPath: binPath})","handlingStrategy":"validation","validationCode":"// validate the binary before constructing the pool VM\nfunc validateJsonnetBinary(path string) error {\n    fi, err := os.Stat(path)\n    if err != nil { return fmt.Errorf(\"jsonnet binary missing: %w\", err) }\n    if fi.IsDir() { return errors.New(\"path is a directory\") }\n    if fi.Mode()&0o111 == 0 { return errors.New(\"jsonnet binary is not executable\") }\n    return nil\n}","typeGuard":null,"tryCatchPattern":"worker, err := vm.pool.puddle.Acquire(ctx)\nif err != nil {\n    var execErr *exec.Error\n    if errors.As(errors.Cause(err), &execErr) {\n        log.Fatalf(\"jsonnet binary %q unusable: %v\", execErr.Name, execErr.Err)\n    }\n    return \"\", errors.Wrap(err, \"jsonnetsecure: acquire\")\n}","preventionTips":["Assert the binary exists and is executable at service startup (fail fast)","Include the jsonnet binary in your container image and verify its architecture","Smoke-test `jsonnet -0` with `{}` in readiness probes","Watch for cgroup pids/memory limits that block fork under load"],"tags":["go","subprocess","exec","container","binary-not-found"],"backgroundTag":"exec-binary-not-found","analyzedSha":"4174065ffb052799890f7480f5360a877a67ffc1","analyzedAt":"2026-09-03T14:52:41.581Z","contentChangedAt":"2026-09-03T14:52:41.581Z","schemaVersion":2},"datasetVersion":"2026-09-10T17:17:09.494Z"}