ory/hydra · critical

newWorker: warm up failed

Error message

newWorker: warm up failed

What it means

After starting the worker process, `newWorker` runs a warm-up evaluation of `{}` through the full stdin/stdout protocol; any failure (write error, stderr message, context timeout) triggers the worker to be destroyed and this wrapped error returned. It means a freshly spawned worker could not complete even a trivial evaluation, so the pool never gets this resource.

Source

Thrown at oryx/jsonnetsecure/jsonnet_pool.go:191

		if err := scanner.Err(); err != nil {
			c <- "ERROR: scan: " + err.Error()
		}
	}
	out := make(chan string, 1)
	go scan(out, stdout, jsonnetOutputLimit)
	errs := make(chan string, 1)
	go scan(errs, stderr, jsonnetErrLimit)

	w := worker{
		cmd:    cmd,
		stdin:  in,
		stdout: out,
		stderr: errs,
	}
	_, err = w.eval(ctx, []byte("{}")) // warm up
	if err != nil {
		w.destroy()
		return worker{}, errors.Wrap(err, "newWorker: warm up failed")
	}

	return w, nil
}

func (w worker) destroy() {
	close(w.stdin)
	w.cmd.Process.Kill()
	w.cmd.Wait()
}

func (w worker) eval(ctx context.Context, processParams []byte) (output string, err error) {
	tracer := trace.SpanFromContext(ctx).TracerProvider().Tracer("")
	ctx, span := tracer.Start(ctx, "jsonnetsecure.worker.eval", trace.WithAttributes(
		semconv.ProcessPID(w.cmd.Process.Pid)))
	defer otelx.End(span, &err)

	// The worker is exclusively acquired, so the schedstat delta over this

View on GitHub (pinned to 4174065ffb)

Solutions

1. Run the child command manually (`jsonnet -0` with `{}` on stdin) to see the real stderr error
2. Check kernel/OS landlock support (Linux 5.13+) and that seccomp/apparmor in your container permits the landlock syscalls
3. Raise timeouts or reduce host load — the eval timeout is a hard 1s in EvaluateAnonymousSnippet
4. Verify the 2GiB virtual memory limit (SetVirtualMemoryLimit) is attainable; containers with strict memory cgroups may kill the child

Example fix

// before: blindly retrying pool acquisition on failure
vm, err := jsonnetsecure.NewProcessPoolVM(opts)
result, err := vm.EvaluateAnonymousSnippet("f.jsonnet", snippet) // newWorker: warm up failed
// after: fall back to the in-process VM when the sandbox can't start (e.g. no landlock)
result, err := vm.EvaluateAnonymousSnippet("f.jsonnet", snippet)
if err != nil && strings.Contains(err.Error(), "warm up failed") {
    result, err = jsonnetsecure.MakeInProcessVM().EvaluateAnonymousSnippet("f.jsonnet", snippet)
}

// precheck: can a trivial eval succeed in this environment?
cmd := exec.Command(binPath, "-0")
cmd.Stdin = strings.NewReader("{}")
var out, errb bytes.Buffer
cmd.Stdout, cmd.Stderr = &out, &errb
if err := cmd.Run(); err != nil {
    return fmt.Errorf("jsonnet smoke test failed: %v: %s", err, errb.String())
}

result, err := vm.EvaluateAnonymousSnippet("conf.jsonnet", snippet)
if err != nil && strings.Contains(err.Error(), "warm up failed") {
    // retry once; transient load can exceed the 1s warm-up budget
    time.Sleep(200 * time.Millisecond)
    result, err = vm.EvaluateAnonymousSnippet("conf.jsonnet", snippet)
}

When it happens

Trigger: Worker creation warm-up `w.eval(ctx, []byte("{}"))` fails: the child wrote to stderr (e.g. landlock sandbox failure, crash), the eval context expired, or the 1s eval timeout hit on a heavily loaded host — all wrapped as `newWorker: warm up failed`.

Common situations: Kernel without landlock support or seccomp blocking landlock syscalls in the container; host so loaded that the warm-up exceeds the 1s eval timeout; child killed by the 2GiB virtual-memory ulimit in constrained environments; slow cold-start in CI.

Related errors

• import not available %v
• failed to write json output
• newWorker: failed to create stdin pipe
• newWorker: failed to create stdout pipe
• newWorker: failed to create stderr pipe

AI-assisted analysis of ory/hydra@4174065ffb (2026-09-03). Data as JSON: /api/errors/d4e6df05e98117c3. Report an issue: GitHub.