cloudflare/pingora · error

failed to build offload runtime

Error message

failed to build offload runtime

What it means

pingora-core lazily builds pools of single-threaded Tokio runtimes for offloading work (TLS handshakes, connection establishment) the first time get_runtime() runs after startup (offload.rs:61-77). Creating each runtime via Builder::new_current_thread().enable_all().build() can fail if the OS cannot provide the resources Tokio needs (I/O/time drivers), and the code .expect()s success, panicking with 'failed to build offload runtime'.

Source

Thrown at pingora-core/src/offload.rs:72

            thread_name,
            shards,
            thread_per_shard,
            pools: OnceCell::new(),
        }
    }

    /// Build every runtime thread in this pool.
    fn init_pools(&self) -> Box<[(Handle, Sender<()>)]> {
        let threads = self.shards * self.thread_per_shard;
        let mut pools = Vec::with_capacity(threads);
        for shard in 0..self.shards {
            for thread in 0..self.thread_per_shard {
                // We use single thread runtimes to reduce the scheduling overhead of multithread
                // tokio runtime, which can be 50% of the on CPU time of the runtimes
                let rt = Builder::new_current_thread()
                    .enable_all()
                    .build()
                    .expect("failed to build offload runtime");
                let handler = rt.handle().clone();
                let (tx, rx) = channel::<()>();
                let thread_name = format!("{} {shard}.{thread}", self.thread_name);
                std::thread::Builder::new()
                    .name(thread_name.clone())
                    .spawn(move || {
                        debug!("{thread_name} started");
                        // the thread that calls block_on() will drive the runtime
                        // rx will return when tx is dropped so this runtime and thread will exit
                        rt.block_on(rx)
                    })
                    .expect("failed to spawn offload runtime thread");
                pools.push((handler, tx));
            }
        }

        pools.into_boxed_slice()
    }

View on GitHub (pinned to 0046038bd4)

Solutions

1. Check and raise file-descriptor limits (ulimit -n) and look for fd leaks: ls /proc/<pid>/fd | wc -l
2. Reduce the pool sizes (fewer pools/threads per pool) to lower fd pressure from epoll handles
3. Disable the offload threadpools (remove the ServerConf fields / don't call set_offload_threadpool) to fall back to inline processing

Example fix

# before
ulimit -n 1024  # runtime creation can fail with EMFILE under load

# after
ulimit -n 65536  # in the service unit or container spec

// Startup probe: can this process still create a Tokio runtime + thread
// before lazy offload pool creation is triggered by the first handshake?
fn offload_probe() -> bool {
    std::panic::catch_unwind(|| {
        let rt = tokio::runtime::Builder::new_current_thread()
            .enable_all()
            .build()
            .expect("probe runtime");
        std::thread::spawn(move || rt.block_on(async {}));
    })
    .is_ok()
}

When it happens

Trigger: Enabling offload threadpools — ServerConf fields downstream_tls_offload_threadpools / upstream_connect_offload_threadpools, or TlsSettings::set_offload_threadpool() — and then reaching the first offloaded TLS handshake or connect under conditions where Tokio runtime creation fails, typically fd exhaustion (EMFILE) or restrictive seccomp/sandboxing.

Common situations: Long-running proxies that leaked file descriptors until creation of new epoll handles fails; containers with tight RLIMIT_NOFILE; hardened/seccomp environments where epoll/timerfd syscalls are filtered; heavy fork/daemonize edge cases.

Related errors

• failed to build tokio runtime for parent signal wait
• failed to build work-stealing Tokio runtime
• failed to build no-steal Tokio runtime worker
• failed to register SIGUSR1 listener
• No tls feature was specified

AI-assisted analysis of cloudflare/pingora@0046038bd4 (2026-08-16). Data as JSON: /api/errors/e75a105ab81e6738. Report an issue: GitHub.