{"record":{"id":"3a710bd8d16f2f08","repo":"cloudflare/pingora","slug":"failed-to-build-no-steal-tokio-runtime-worker","errorCode":null,"errorMessage":"failed to build no-steal Tokio runtime worker","messagePattern":"failed to build no-steal Tokio runtime worker","errorType":"panic","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"pingora-runtime/src/lib.rs","lineNumber":632,"sourceCode":"            name: name.to_string(),\n            blocking_opts,\n            runtime_opts,\n            pools: Arc::new(OnceCell::new()),\n            controls: OnceCell::new(),\n        }\n    }\n\n    fn init_pools(&self) -> (Box<[Handle]>, Vec<Control>) {\n        let mut pools = Vec::with_capacity(self.threads);\n        let mut controls = Vec::with_capacity(self.threads);\n        for _ in 0..self.threads {\n            let mut builder = Builder::new_current_thread();\n            builder.enable_all();\n            apply_blocking_opts(&mut builder, &self.blocking_opts);\n            apply_metrics_opts(&mut builder, &self.runtime_opts.metrics);\n            let rt = builder\n                .build()\n                .expect(\"failed to build no-steal Tokio runtime worker\");\n            let handler = rt.handle().clone();\n            let (tx, rx) = channel::<Duration>();\n            let pools_ref = self.pools.clone();\n            let join = std::thread::Builder::new()\n                .name(self.name.clone())\n                .spawn(move || {\n                    CURRENT_HANDLE.get_or(|| pools_ref);\n                    if let Ok(timeout) = rt.block_on(rx) {\n                        rt.shutdown_timeout(timeout);\n                    } // else Err(_): tx is dropped, just exit\n                })\n                .unwrap();\n            pools.push(handler);\n            controls.push((tx, join));\n        }\n\n        (pools.into_boxed_slice(), controls)\n    }","sourceCodeStart":614,"sourceCodeEnd":650,"githubUrl":"https://github.com/cloudflare/pingora/blob/0046038bd402bc82912da862dadf9a479f31e9f1/pingora-runtime/src/lib.rs#L614-L650","documentation":"pingora's no-steal runtime mode creates N independent current-thread runtimes plus N OS threads (one pool per configured thread). This expect fires if any single runtime build fails during init_pools — practically when the process can no longer create threads or driver resources (fds/epoll) under its limits, partway through creating the pool.","triggerScenarios":"Configuring the runtime with work stealing disabled and N threads where building the Nth current_thread runtime or spawning its worker thread fails: RLIMIT_NPROC/pids.max exhausted, fd limit reached (each per-thread runtime registers its own driver fds), or memory/thread-stack exhaustion.","commonSituations":"Tight pid/fd quotas in containers with many requested threads; thread counts copied from a work-stealing config into a constrained cgroup; multiple embedded runtimes in one process.","solutions":["Reduce the thread count for the no-steal runtime to fit the pid/fd budget","Raise pids.max and RLIMIT_NOFILE — remember each no-steal thread consumes driver fds of its own","Check dmesg/journald at the same timestamp for thread-spawn or 'cannot allocate memory' errors","Re-tune per mode: no-steal spawns one runtime + one thread per configured worker, so budgets differ from work-stealing"],"exampleFix":null,"handlingStrategy":"validation","validationCode":"// No-steal mode: N runtimes + N threads. Probe that budget first.\nfn can_build_n_single_thread_runtimes(n: usize) -> bool {\n    (0..n).all(|_| {\n        tokio::runtime::Builder::new_current_thread()\n            .enable_all()\n            .build()\n            .is_ok()\n    })\n}","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Budget one runtime plus one OS thread (and its driver fds) per configured no-steal worker","Lower thread counts in constrained cgroups rather than copying work-stealing configs","Monitor pids.max and RLIMIT_NOFILE headroom; fail deploy checks when below threads + margin","Check dmesg for thread-spawn failures when startup dies partway through pool creation"],"tags":["rust","pingora","tokio","runtime","threads","resource-limits","startup","panic"],"backgroundTag":"runtime-initialization-failed","analyzedSha":"0046038bd402bc82912da862dadf9a479f31e9f1","analyzedAt":"2026-08-16T21:33:22.341Z","schemaVersion":2},"datasetVersion":"2026-08-16T23:17:17.608Z"}