{"record":{"id":"e9217c9781881330","repo":"cloudflare/pingora","slug":"failed-to-build-work-stealing-tokio-runtime","errorCode":null,"errorMessage":"failed to build work-stealing Tokio runtime","messagePattern":"failed to build work-stealing Tokio runtime","errorType":"panic","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"pingora-runtime/src/lib.rs","lineNumber":506,"sourceCode":"                        return Runtime::Steal {\n                            runtime,\n                            dial9_guard: Some(guard),\n                        };\n                    }\n                    Err(e) => {\n                        log::warn!(\n                            \"failed to initialize dial9 runtime telemetry for {runtime_name}: {e}\"\n                        );\n                        builder = self.build_work_stealing_tokio_builder();\n                        None\n                    }\n                }\n            } else {\n                None\n            };\n            let runtime = builder\n                .build()\n                .expect(\"failed to build work-stealing Tokio runtime\");\n            Runtime::Steal {\n                runtime,\n                #[cfg(feature = \"dial9\")]\n                dial9_guard,\n            }\n        } else {\n            #[cfg(feature = \"dial9\")]\n            if self.runtime_opts.dial9.is_some() {\n                log::warn!(\"dial9 runtime telemetry is ignored when work stealing is disabled\");\n            }\n            Runtime::NoSteal(NoStealRuntime::new(\n                self.threads,\n                &self.name,\n                self.blocking_pool_opts,\n                self.runtime_opts,\n            ))\n        }\n    }","sourceCodeStart":488,"sourceCodeEnd":524,"githubUrl":"https://github.com/cloudflare/pingora/blob/0046038bd402bc82912da862dadf9a479f31e9f1/pingora-runtime/src/lib.rs#L488-L524","documentation":"Building pingora's multi-threaded (work-stealing) tokio runtime failed at startup. Runtime construction returns Err when worker threads cannot be spawned (thread/pid limits, memory), when driver resources cannot register with the OS (epoll/eventfd), or when thread/stack settings are invalid for the machine. Note this expect sits after the optional dial9 telemetry init, which already falls back to a plain builder on telemetry errors — so the failure comes from the builder/build itself.","triggerScenarios":"Server startup with work stealing enabled where builder.build() returns Err: RLIMIT_NPROC or cgroup pids.max exhausted while spawning workers, a huge configured thread count, thread-stack/memory exhaustion, or a sandbox blocking epoll/eventfd.","commonSituations":"Kubernetes containers with low pids limits; threads configured higher than the environment allows; supervisors with hard NPROC; seccomp profiles blocking epoll_create1/eventfd; several runtimes in one process.","solutions":["Check thread headroom: `ulimit -u`, `/sys/fs/cgroup/pids.max` (v2) — raise the limit or lower pingora's thread count","Reduce the configured worker threads in the server conf to fit the environment","If stacks can't allocate, reduce thread stack size or thread count (check `ulimit -s`, dmesg for 'cannot allocate memory')","Allow epoll_create1/eventfd in the sandbox/seccomp policy — tokio drivers require them"],"exampleFix":null,"handlingStrategy":"validation","validationCode":"// Probe whether the environment can host the runtime before server startup\nfn can_build_work_stealing_runtime(workers: usize) -> bool {\n    tokio::runtime::Builder::new_multi_thread()\n        .worker_threads(workers)\n        .enable_all()\n        .build()\n        .is_ok()\n}","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Set pingora thread counts explicitly to fit container pid budgets (e.g. align with CPU limits)","Check ulimit -u, ulimit -n and cgroup pids.max in the deploy environment before shipping","Load-test startup inside the production container image in CI","Keep one runtime per process; don't embed pingora runtimes inside other runtimes"],"tags":["rust","pingora","tokio","runtime","threads","resource-limits","startup","panic"],"backgroundTag":"runtime-initialization-failed","analyzedSha":"0046038bd402bc82912da862dadf9a479f31e9f1","analyzedAt":"2026-08-16T21:33:22.341Z","schemaVersion":2},"datasetVersion":"2026-08-16T23:17:17.608Z"}