zellij-org/zellij · critical

failed to spawn forward-timeout driver thread

Error message

failed to spawn forward-timeout driver thread

What it means

Spawns the driver thread that keeps the forward-timeout runtime alive by parking on block_on(pending()). std::thread::Builder::spawn returns io::Error when the OS refuses a new thread - RLIMIT_NPROC or cgroup pids.max reached, or insufficient memory for the thread stack - and this expect turns that into an immediate panic at first runtime use, killing the client thread.

Source

Thrown at zellij-client/src/stdin_ansi_parser.rs:914

static FORWARD_TIMEOUT_RUNTIME: OnceLock<Arc<tokio::runtime::Runtime>> = OnceLock::new();

pub fn forward_timeout_runtime() -> &'static Arc<tokio::runtime::Runtime> {
    FORWARD_TIMEOUT_RUNTIME.get_or_init(|| {
        let rt = tokio::runtime::Builder::new_current_thread()
            .enable_time()
            .build()
            .expect("failed to build forward-timeout runtime");
        let rt = Arc::new(rt);
        let rt_for_driver = rt.clone();
        // `block_on(pending())` keeps the executor loop alive forever
        // on this thread; spawned timer tasks are polled as they
        // become ready (on spawn, on wake from the time driver).
        std::thread::Builder::new()
            .name("zellij-client-forward-timeout".into())
            .spawn(move || {
                rt_for_driver.block_on(std::future::pending::<()>());
            })
            .expect("failed to spawn forward-timeout driver thread");
        rt
    })
}

/// Spawn a timer task that closes a forward slot after `deadline` and
/// invokes `on_timeout(token, reply_bytes)` with whatever the slot
/// accumulated. Token-guard idempotent: if the barrier (or a
/// replacement forward) has already cleared the slot by the time the
/// timer wakes, `close_forward_on_timeout(token)` returns `None` and
/// `on_timeout` is never called — no explicit cancellation path
/// required.
///
/// Extracted as a free function so tests can drive it against a
/// `tokio::time::pause()`-backed paused runtime without instantiating
/// the full client.
pub fn schedule_forward_timeout<F>(
    runtime: &tokio::runtime::Handle,
    parser: Arc<Mutex<StdinAnsiParser>>,

View on GitHub (pinned to 98a0837077)

Solutions

1. Check `ulimit -u` and the cgroup pids.max; raise them for the zellij client
2. Reduce threads already held by the process (close stale sessions/clients)
3. Free memory or lower per-thread stack size and restart the client
4. Restart the process group if leaked processes consume the limit

Example fix

// before
std::thread::Builder::new()
    .name("zellij-client-forward-timeout".into())
    .spawn(move || { /* ... */ })
    .expect("failed to spawn forward-timeout driver thread");

// after - surface a contextual error instead of a bare panic
let handle = std::thread::Builder::new()
    .name("zellij-client-forward-timeout".into())
    .spawn(move || { /* ... */ })
    .context("could not spawn forward-timeout driver thread (thread/memory limit?)")
    .fatal();

let mut handle = None;
for attempt in 0..3 {
    match std::thread::Builder::new().name("timer-driver".into()).spawn(driver_fn()) {
        Ok(h) => {
            handle = Some(h);
            break;
        }
        Err(e) if e.kind() == std::io::ErrorKind::WouldBlock => {
            std::thread::sleep(std::time::Duration::from_millis(200 * (attempt as u64 + 1)));
        }
        Err(e) => {
            log::error!("driver thread spawn failed hard: {e}");
            break;
        }
    }
}

When it happens

Trigger: The get_or_init of FORWARD_TIMEOUT_RUNTIME running on a host at its thread or memory limit at the moment the first forward timeout is scheduled.

Common situations: Machines with aggressive cgroup pids limits (containers, CI runners); thread leaks in the client; fork-bombed or heavily loaded hosts.

Understand the failure class

• Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

• Failed to create tokio runtime
• failed to build forward-timeout runtime
• Could not find editor pane to replace - is no pane focused?
• Could not find editor pane to replace
• failed to find active pane id for client {client_id}

AI-assisted analysis of zellij-org/zellij@98a0837077 (2026-08-16). Data as JSON: /api/errors/bb9acc460c38b14c. Report an issue: GitHub.