What causes "failed to build no-steal Tokio runtime worker" in cloudflare/pingora?

Reduce the thread count for the no-steal runtime to fit the pid/fd budget

failed to build no-steal Tokio runtime worker

Error message

failed to build no-steal Tokio runtime worker

What it means

pingora's no-steal runtime mode creates N independent current-thread runtimes plus N OS threads (one pool per configured thread). This expect fires if any single runtime build fails during init_pools — practically when the process can no longer create threads or driver resources (fds/epoll) under its limits, partway through creating the pool.

Source

Thrown at pingora-runtime/src/lib.rs:632

            name: name.to_string(),
            blocking_opts,
            runtime_opts,
            pools: Arc::new(OnceCell::new()),
            controls: OnceCell::new(),
        }
    }

    fn init_pools(&self) -> (Box<[Handle]>, Vec<Control>) {
        let mut pools = Vec::with_capacity(self.threads);
        let mut controls = Vec::with_capacity(self.threads);
        for _ in 0..self.threads {
            let mut builder = Builder::new_current_thread();
            builder.enable_all();
            apply_blocking_opts(&mut builder, &self.blocking_opts);
            apply_metrics_opts(&mut builder, &self.runtime_opts.metrics);
            let rt = builder
                .build()
                .expect("failed to build no-steal Tokio runtime worker");
            let handler = rt.handle().clone();
            let (tx, rx) = channel::<Duration>();
            let pools_ref = self.pools.clone();
            let join = std::thread::Builder::new()
                .name(self.name.clone())
                .spawn(move || {
                    CURRENT_HANDLE.get_or(|| pools_ref);
                    if let Ok(timeout) = rt.block_on(rx) {
                        rt.shutdown_timeout(timeout);
                    } // else Err(_): tx is dropped, just exit
                })
                .unwrap();
            pools.push(handler);
            controls.push((tx, join));
        }

        (pools.into_boxed_slice(), controls)
    }

View on GitHub (pinned to 0046038bd4)

Solutions

Reduce the thread count for the no-steal runtime to fit the pid/fd budget
Raise pids.max and RLIMIT_NOFILE — remember each no-steal thread consumes driver fds of its own
Check dmesg/journald at the same timestamp for thread-spawn or 'cannot allocate memory' errors
Re-tune per mode: no-steal spawns one runtime + one thread per configured worker, so budgets differ from work-stealing

Defensive patterns

Strategy: validation

Validate before calling

// No-steal mode: N runtimes + N threads. Probe that budget first.
fn can_build_n_single_thread_runtimes(n: usize) -> bool {
    (0..n).all(|_| {
        tokio::runtime::Builder::new_current_thread()
            .enable_all()
            .build()
            .is_ok()
    })
}

Prevention

Budget one runtime plus one OS thread (and its driver fds) per configured no-steal worker
Lower thread counts in constrained cgroups rather than copying work-stealing configs
Monitor pids.max and RLIMIT_NOFILE headroom; fail deploy checks when below threads + margin
Check dmesg for thread-spawn failures when startup dies partway through pool creation

When it happens

Trigger: Configuring the runtime with work stealing disabled and N threads where building the Nth current_thread runtime or spawning its worker thread fails: RLIMIT_NPROC/pids.max exhausted, fd limit reached (each per-thread runtime registers its own driver fds), or memory/thread-stack exhaustion.

Common situations: Tight pid/fd quotas in containers with many requested threads; thread counts copied from a work-stealing config into a constrained cgroup; multiple embedded runtimes in one process.

Related errors

AI-assisted analysis of cloudflare/pingora@0046038bd4 (2026-08-16). Data as JSON: /api/errors/3a710bd8d16f2f08. Report an issue: GitHub.