{"record":{"id":"e75a105ab81e6738","repo":"cloudflare/pingora","slug":"failed-to-build-offload-runtime","errorCode":null,"errorMessage":"failed to build offload runtime","messagePattern":"failed to build offload runtime","errorType":"panic","errorClass":null,"httpStatus":null,"severity":"error","filePath":"pingora-core/src/offload.rs","lineNumber":72,"sourceCode":"            thread_name,\n            shards,\n            thread_per_shard,\n            pools: OnceCell::new(),\n        }\n    }\n\n    /// Build every runtime thread in this pool.\n    fn init_pools(&self) -> Box<[(Handle, Sender<()>)]> {\n        let threads = self.shards * self.thread_per_shard;\n        let mut pools = Vec::with_capacity(threads);\n        for shard in 0..self.shards {\n            for thread in 0..self.thread_per_shard {\n                // We use single thread runtimes to reduce the scheduling overhead of multithread\n                // tokio runtime, which can be 50% of the on CPU time of the runtimes\n                let rt = Builder::new_current_thread()\n                    .enable_all()\n                    .build()\n                    .expect(\"failed to build offload runtime\");\n                let handler = rt.handle().clone();\n                let (tx, rx) = channel::<()>();\n                let thread_name = format!(\"{} {shard}.{thread}\", self.thread_name);\n                std::thread::Builder::new()\n                    .name(thread_name.clone())\n                    .spawn(move || {\n                        debug!(\"{thread_name} started\");\n                        // the thread that calls block_on() will drive the runtime\n                        // rx will return when tx is dropped so this runtime and thread will exit\n                        rt.block_on(rx)\n                    })\n                    .expect(\"failed to spawn offload runtime thread\");\n                pools.push((handler, tx));\n            }\n        }\n\n        pools.into_boxed_slice()\n    }","sourceCodeStart":54,"sourceCodeEnd":90,"githubUrl":"https://github.com/cloudflare/pingora/blob/0046038bd402bc82912da862dadf9a479f31e9f1/pingora-core/src/offload.rs#L54-L90","documentation":"pingora-core lazily builds pools of single-threaded Tokio runtimes for offloading work (TLS handshakes, connection establishment) the first time get_runtime() runs after startup (offload.rs:61-77). Creating each runtime via Builder::new_current_thread().enable_all().build() can fail if the OS cannot provide the resources Tokio needs (I/O/time drivers), and the code .expect()s success, panicking with 'failed to build offload runtime'.","triggerScenarios":"Enabling offload threadpools — ServerConf fields downstream_tls_offload_threadpools / upstream_connect_offload_threadpools, or TlsSettings::set_offload_threadpool() — and then reaching the first offloaded TLS handshake or connect under conditions where Tokio runtime creation fails, typically fd exhaustion (EMFILE) or restrictive seccomp/sandboxing.","commonSituations":"Long-running proxies that leaked file descriptors until creation of new epoll handles fails; containers with tight RLIMIT_NOFILE; hardened/seccomp environments where epoll/timerfd syscalls are filtered; heavy fork/daemonize edge cases.","solutions":["Check and raise file-descriptor limits (ulimit -n) and look for fd leaks: ls /proc/<pid>/fd | wc -l","Reduce the pool sizes (fewer pools/threads per pool) to lower fd pressure from epoll handles","Disable the offload threadpools (remove the ServerConf fields / don't call set_offload_threadpool) to fall back to inline processing"],"exampleFix":"# before\nulimit -n 1024  # runtime creation can fail with EMFILE under load\n\n# after\nulimit -n 65536  # in the service unit or container spec","handlingStrategy":"validation","validationCode":"// Startup probe: can this process still create a Tokio runtime + thread\n// before lazy offload pool creation is triggered by the first handshake?\nfn offload_probe() -> bool {\n    std::panic::catch_unwind(|| {\n        let rt = tokio::runtime::Builder::new_current_thread()\n            .enable_all()\n            .build()\n            .expect(\"probe runtime\");\n        std::thread::spawn(move || rt.block_on(async {}));\n    })\n    .is_ok()\n}","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Set generous RLIMIT_NOFILE in the service unit/container and monitor /proc/<pid>/fd growth for leaks","Size offload pools conservatively (shards x threads) relative to the machine's fd budget","Alert on fd usage well before the hard limit so runtime creation never runs dry"],"tags":["rust","tokio","runtime","resource-limits","fd-exhaustion","pingora"],"backgroundTag":"tokio-runtime-creation-failed","analyzedSha":"0046038bd402bc82912da862dadf9a479f31e9f1","analyzedAt":"2026-08-16T21:33:22.341Z","schemaVersion":2},"datasetVersion":"2026-08-16T23:17:17.608Z"}