{"record":{"id":"d4e6df05e98117c3","repo":"ory/hydra","slug":"newworker-warm-up-failed","errorCode":null,"errorMessage":"newWorker: warm up failed","messagePattern":"newWorker: warm up failed","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"oryx/jsonnetsecure/jsonnet_pool.go","lineNumber":191,"sourceCode":"\t\tif err := scanner.Err(); err != nil {\n\t\t\tc <- \"ERROR: scan: \" + err.Error()\n\t\t}\n\t}\n\tout := make(chan string, 1)\n\tgo scan(out, stdout, jsonnetOutputLimit)\n\terrs := make(chan string, 1)\n\tgo scan(errs, stderr, jsonnetErrLimit)\n\n\tw := worker{\n\t\tcmd:    cmd,\n\t\tstdin:  in,\n\t\tstdout: out,\n\t\tstderr: errs,\n\t}\n\t_, err = w.eval(ctx, []byte(\"{}\")) // warm up\n\tif err != nil {\n\t\tw.destroy()\n\t\treturn worker{}, errors.Wrap(err, \"newWorker: warm up failed\")\n\t}\n\n\treturn w, nil\n}\n\nfunc (w worker) destroy() {\n\tclose(w.stdin)\n\tw.cmd.Process.Kill()\n\tw.cmd.Wait()\n}\n\nfunc (w worker) eval(ctx context.Context, processParams []byte) (output string, err error) {\n\ttracer := trace.SpanFromContext(ctx).TracerProvider().Tracer(\"\")\n\tctx, span := tracer.Start(ctx, \"jsonnetsecure.worker.eval\", trace.WithAttributes(\n\t\tsemconv.ProcessPID(w.cmd.Process.Pid)))\n\tdefer otelx.End(span, &err)\n\n\t// The worker is exclusively acquired, so the schedstat delta over this","sourceCodeStart":173,"sourceCodeEnd":209,"githubUrl":"https://github.com/ory/hydra/blob/4174065ffb052799890f7480f5360a877a67ffc1/oryx/jsonnetsecure/jsonnet_pool.go#L173-L209","documentation":"After starting the worker process, `newWorker` runs a warm-up evaluation of `{}` through the full stdin/stdout protocol; any failure (write error, stderr message, context timeout) triggers the worker to be destroyed and this wrapped error returned. It means a freshly spawned worker could not complete even a trivial evaluation, so the pool never gets this resource.","triggerScenarios":"Worker creation warm-up `w.eval(ctx, []byte(\"{}\"))` fails: the child wrote to stderr (e.g. landlock sandbox failure, crash), the eval context expired, or the 1s eval timeout hit on a heavily loaded host — all wrapped as `newWorker: warm up failed`.","commonSituations":"Kernel without landlock support or seccomp blocking landlock syscalls in the container; host so loaded that the warm-up exceeds the 1s eval timeout; child killed by the 2GiB virtual-memory ulimit in constrained environments; slow cold-start in CI.","solutions":["Run the child command manually (`jsonnet -0` with `{}` on stdin) to see the real stderr error","Check kernel/OS landlock support (Linux 5.13+) and that seccomp/apparmor in your container permits the landlock syscalls","Raise timeouts or reduce host load — the eval timeout is a hard 1s in EvaluateAnonymousSnippet","Verify the 2GiB virtual memory limit (SetVirtualMemoryLimit) is attainable; containers with strict memory cgroups may kill the child"],"exampleFix":"// before: blindly retrying pool acquisition on failure\nvm, err := jsonnetsecure.NewProcessPoolVM(opts)\nresult, err := vm.EvaluateAnonymousSnippet(\"f.jsonnet\", snippet) // newWorker: warm up failed\n// after: fall back to the in-process VM when the sandbox can't start (e.g. no landlock)\nresult, err := vm.EvaluateAnonymousSnippet(\"f.jsonnet\", snippet)\nif err != nil && strings.Contains(err.Error(), \"warm up failed\") {\n    result, err = jsonnetsecure.MakeInProcessVM().EvaluateAnonymousSnippet(\"f.jsonnet\", snippet)\n}","handlingStrategy":"retry","validationCode":"// precheck: can a trivial eval succeed in this environment?\ncmd := exec.Command(binPath, \"-0\")\ncmd.Stdin = strings.NewReader(\"{}\")\nvar out, errb bytes.Buffer\ncmd.Stdout, cmd.Stderr = &out, &errb\nif err := cmd.Run(); err != nil {\n    return fmt.Errorf(\"jsonnet smoke test failed: %v: %s\", err, errb.String())\n}","typeGuard":null,"tryCatchPattern":"result, err := vm.EvaluateAnonymousSnippet(\"conf.jsonnet\", snippet)\nif err != nil && strings.Contains(err.Error(), \"warm up failed\") {\n    // retry once; transient load can exceed the 1s warm-up budget\n    time.Sleep(200 * time.Millisecond)\n    result, err = vm.EvaluateAnonymousSnippet(\"conf.jsonnet\", snippet)\n}","preventionTips":["Verify landlock support (Linux >= 5.13) and seccomp/apparmor profiles that don't block it","Smoke-test worker startup at boot, not on first user request","Size hosts so a trivial eval finishes well under the 1s timeout","Check the 2GiB virtual-memory ulimit is attainable inside restrictive cgroups"],"tags":["go","subprocess","sandbox","landlock","timeout"],"backgroundTag":"worker-warmup-failed","analyzedSha":"4174065ffb052799890f7480f5360a877a67ffc1","analyzedAt":"2026-09-03T14:52:41.581Z","contentChangedAt":"2026-09-03T14:52:41.581Z","schemaVersion":2},"datasetVersion":"2026-09-10T17:17:09.494Z"}