What causes "failed to spawn offload runtime thread" in cloudflare/pingora?

Raise the thread budget: container pids limit / RLIMIT_NPROC / kernel threads-max, or add memory

failed to spawn offload runtime thread

Error message

failed to spawn offload runtime thread

What it means

Companion panic in offload.rs:84: after each offload Tokio runtime is built, std::thread::Builder::spawn() starts the thread that drives it, and the result is .expect()ed. If the OS refuses to create the thread (thread/memory limits, cgroup pids.max, OOM), the process panics with 'failed to spawn offload runtime thread'. Because pools are created lazily on first use, this typically appears under load, not at boot.

Source

Thrown at pingora-core/src/offload.rs:84

            for thread in 0..self.thread_per_shard {
                // We use single thread runtimes to reduce the scheduling overhead of multithread
                // tokio runtime, which can be 50% of the on CPU time of the runtimes
                let rt = Builder::new_current_thread()
                    .enable_all()
                    .build()
                    .expect("failed to build offload runtime");
                let handler = rt.handle().clone();
                let (tx, rx) = channel::<()>();
                let thread_name = format!("{} {shard}.{thread}", self.thread_name);
                std::thread::Builder::new()
                    .name(thread_name.clone())
                    .spawn(move || {
                        debug!("{thread_name} started");
                        // the thread that calls block_on() will drive the runtime
                        // rx will return when tx is dropped so this runtime and thread will exit
                        rt.block_on(rx)
                    })
                    .expect("failed to spawn offload runtime thread");
                pools.push((handler, tx));
            }
        }

        pools.into_boxed_slice()
    }

    /// Return the runtime for `hash`.
    ///
    /// `hash` selects the shard. A runtime within that shard is chosen randomly
    /// to spread work across `thread_per_shard` runtimes.
    pub fn get_runtime(&self, hash: u64) -> &Handle {
        let mut rng = rand::thread_rng();

        // choose a shard based on hash and a random thread with in that shard
        // e.g. say thread_per_shard=2, shard 1 thread 1 is 1 * 2 + 1 = 3
        // [[th0, th1], [th2, th3], ...]
        let shard = hash as usize % self.shards;

View on GitHub (pinned to 0046038bd4)

Solutions

Raise the thread budget: container pids limit / RLIMIT_NPROC / kernel threads-max, or add memory
Shrink the offload pools: fewer shards and threads_per_shard in ServerConf / set_offload_threadpool
Audit total thread usage (cat /proc/<pid>/status | grep Threads) and other runtime pools in the same process

Example fix

# before (docker): threads exhausted by offload pools
--pids-limit 64

# after
--pids-limit 512  # or reduce downstream_tls_offload_thread_per_pool in the yaml

Defensive patterns

Strategy: validation

Validate before calling

// Startup probe: verify the thread budget before offload pools spin up
fn thread_budget_probe(want: usize) -> bool {
    let mut ok = true;
    for _ in 0..want {
        if std::thread::Builder::new().spawn(|| {}).is_err() {
            ok = false;
            break;
        }
    }
    ok
}

Prevention

Set container pids limits and RLIMIT_NPROC with headroom for shards x threads_per_shard plus worker threads
Track /proc/<pid>/status Threads in metrics and alert before the ceiling
Prefer fewer, larger offload pools when instance memory is small

When it happens

Trigger: Configuring offload threadpools (downstream_tls_offload_threadpools x per-pool threads, or set_offload_threadpool) and hitting a thread-creation failure on first TLS handshake: cgroup pids.max exhausted, RLIMIT_NPROC/threads-max reached, or memory too low to map thread stacks.

Common situations: Containers with a low pids limit and other threads (per-connection runtimes, workers) already counted against it; sizing shards x threads_per_shard too aggressively on small instances; memory pressure during traffic spikes.

Related errors

AI-assisted analysis of cloudflare/pingora@0046038bd4 (2026-08-16). Data as JSON: /api/errors/26d33434390af67d. Report an issue: GitHub.