{"record":{"id":"26d33434390af67d","repo":"cloudflare/pingora","slug":"failed-to-spawn-offload-runtime-thread","errorCode":null,"errorMessage":"failed to spawn offload runtime thread","messagePattern":"failed to spawn offload runtime thread","errorType":"panic","errorClass":null,"httpStatus":null,"severity":"error","filePath":"pingora-core/src/offload.rs","lineNumber":84,"sourceCode":"            for thread in 0..self.thread_per_shard {\n                // We use single thread runtimes to reduce the scheduling overhead of multithread\n                // tokio runtime, which can be 50% of the on CPU time of the runtimes\n                let rt = Builder::new_current_thread()\n                    .enable_all()\n                    .build()\n                    .expect(\"failed to build offload runtime\");\n                let handler = rt.handle().clone();\n                let (tx, rx) = channel::<()>();\n                let thread_name = format!(\"{} {shard}.{thread}\", self.thread_name);\n                std::thread::Builder::new()\n                    .name(thread_name.clone())\n                    .spawn(move || {\n                        debug!(\"{thread_name} started\");\n                        // the thread that calls block_on() will drive the runtime\n                        // rx will return when tx is dropped so this runtime and thread will exit\n                        rt.block_on(rx)\n                    })\n                    .expect(\"failed to spawn offload runtime thread\");\n                pools.push((handler, tx));\n            }\n        }\n\n        pools.into_boxed_slice()\n    }\n\n    /// Return the runtime for `hash`.\n    ///\n    /// `hash` selects the shard. A runtime within that shard is chosen randomly\n    /// to spread work across `thread_per_shard` runtimes.\n    pub fn get_runtime(&self, hash: u64) -> &Handle {\n        let mut rng = rand::thread_rng();\n\n        // choose a shard based on hash and a random thread with in that shard\n        // e.g. say thread_per_shard=2, shard 1 thread 1 is 1 * 2 + 1 = 3\n        // [[th0, th1], [th2, th3], ...]\n        let shard = hash as usize % self.shards;","sourceCodeStart":66,"sourceCodeEnd":102,"githubUrl":"https://github.com/cloudflare/pingora/blob/0046038bd402bc82912da862dadf9a479f31e9f1/pingora-core/src/offload.rs#L66-L102","documentation":"Companion panic in offload.rs:84: after each offload Tokio runtime is built, std::thread::Builder::spawn() starts the thread that drives it, and the result is .expect()ed. If the OS refuses to create the thread (thread/memory limits, cgroup pids.max, OOM), the process panics with 'failed to spawn offload runtime thread'. Because pools are created lazily on first use, this typically appears under load, not at boot.","triggerScenarios":"Configuring offload threadpools (downstream_tls_offload_threadpools x per-pool threads, or set_offload_threadpool) and hitting a thread-creation failure on first TLS handshake: cgroup pids.max exhausted, RLIMIT_NPROC/threads-max reached, or memory too low to map thread stacks.","commonSituations":"Containers with a low pids limit and other threads (per-connection runtimes, workers) already counted against it; sizing shards x threads_per_shard too aggressively on small instances; memory pressure during traffic spikes.","solutions":["Raise the thread budget: container pids limit / RLIMIT_NPROC / kernel threads-max, or add memory","Shrink the offload pools: fewer shards and threads_per_shard in ServerConf / set_offload_threadpool","Audit total thread usage (cat /proc/<pid>/status | grep Threads) and other runtime pools in the same process"],"exampleFix":"# before (docker): threads exhausted by offload pools\n--pids-limit 64\n\n# after\n--pids-limit 512  # or reduce downstream_tls_offload_thread_per_pool in the yaml","handlingStrategy":"validation","validationCode":"// Startup probe: verify the thread budget before offload pools spin up\nfn thread_budget_probe(want: usize) -> bool {\n    let mut ok = true;\n    for _ in 0..want {\n        if std::thread::Builder::new().spawn(|| {}).is_err() {\n            ok = false;\n            break;\n        }\n    }\n    ok\n}","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Set container pids limits and RLIMIT_NPROC with headroom for shards x threads_per_shard plus worker threads","Track /proc/<pid>/status Threads in metrics and alert before the ceiling","Prefer fewer, larger offload pools when instance memory is small"],"tags":["rust","thread","os-limits","resource-exhaustion","pingora"],"backgroundTag":"thread-creation-failed","analyzedSha":"0046038bd402bc82912da862dadf9a479f31e9f1","analyzedAt":"2026-08-16T21:33:22.341Z","schemaVersion":2},"datasetVersion":"2026-08-16T23:17:17.608Z"}