What causes "failed to build work-stealing Tokio runtime" in cloudflare/pingora?

Check thread headroom: `ulimit -u`, `/sys/fs/cgroup/pids.max` (v2) — raise the limit or lower pingora's thread count

failed to build work-stealing Tokio runtime

Error message

failed to build work-stealing Tokio runtime

What it means

Building pingora's multi-threaded (work-stealing) tokio runtime failed at startup. Runtime construction returns Err when worker threads cannot be spawned (thread/pid limits, memory), when driver resources cannot register with the OS (epoll/eventfd), or when thread/stack settings are invalid for the machine. Note this expect sits after the optional dial9 telemetry init, which already falls back to a plain builder on telemetry errors — so the failure comes from the builder/build itself.

Source

Thrown at pingora-runtime/src/lib.rs:506

                        return Runtime::Steal {
                            runtime,
                            dial9_guard: Some(guard),
                        };
                    }
                    Err(e) => {
                        log::warn!(
                            "failed to initialize dial9 runtime telemetry for {runtime_name}: {e}"
                        );
                        builder = self.build_work_stealing_tokio_builder();
                        None
                    }
                }
            } else {
                None
            };
            let runtime = builder
                .build()
                .expect("failed to build work-stealing Tokio runtime");
            Runtime::Steal {
                runtime,
                #[cfg(feature = "dial9")]
                dial9_guard,
            }
        } else {
            #[cfg(feature = "dial9")]
            if self.runtime_opts.dial9.is_some() {
                log::warn!("dial9 runtime telemetry is ignored when work stealing is disabled");
            }
            Runtime::NoSteal(NoStealRuntime::new(
                self.threads,
                &self.name,
                self.blocking_pool_opts,
                self.runtime_opts,
            ))
        }
    }

View on GitHub (pinned to 0046038bd4)

Solutions

Check thread headroom: `ulimit -u`, `/sys/fs/cgroup/pids.max` (v2) — raise the limit or lower pingora's thread count
Reduce the configured worker threads in the server conf to fit the environment
If stacks can't allocate, reduce thread stack size or thread count (check `ulimit -s`, dmesg for 'cannot allocate memory')
Allow epoll_create1/eventfd in the sandbox/seccomp policy — tokio drivers require them

Defensive patterns

Strategy: validation

Validate before calling

// Probe whether the environment can host the runtime before server startup
fn can_build_work_stealing_runtime(workers: usize) -> bool {
    tokio::runtime::Builder::new_multi_thread()
        .worker_threads(workers)
        .enable_all()
        .build()
        .is_ok()
}

Prevention

Set pingora thread counts explicitly to fit container pid budgets (e.g. align with CPU limits)
Check ulimit -u, ulimit -n and cgroup pids.max in the deploy environment before shipping
Load-test startup inside the production container image in CI
Keep one runtime per process; don't embed pingora runtimes inside other runtimes

When it happens

Trigger: Server startup with work stealing enabled where builder.build() returns Err: RLIMIT_NPROC or cgroup pids.max exhausted while spawning workers, a huge configured thread count, thread-stack/memory exhaustion, or a sandbox blocking epoll/eventfd.

Common situations: Kubernetes containers with low pids limits; threads configured higher than the environment allows; supervisors with hard NPROC; seccomp profiles blocking epoll_create1/eventfd; several runtimes in one process.

Related errors

AI-assisted analysis of cloudflare/pingora@0046038bd4 (2026-08-16). Data as JSON: /api/errors/e9217c9781881330. Report an issue: GitHub.