facebook/flow · critical

failed to spawn flow_server_main thread

Error message

failed to spawn flow_server_main thread

What it means

run_server spawns the dedicated flow_server_main thread with a large fixed stack (SERVER_MAIN_THREAD_STACK_SIZE, needed because the type checker recurses deeply) and expects the spawn to succeed (rust_port/crates/flow_server/src/server.rs:570-580). std::thread::Builder::spawn fails when the OS refuses to create the thread — almost always resource limits, since reserving a large stack plus one more thread crosses a cgroup/rlimit boundary. The expect crashes the launcher immediately; panics inside the thread are handled separately via catch_unwind.

Source

Thrown at rust_port/crates/flow_server/src/server.rs:579

    };
    flow_event_logger::worker_exception(&data);
}

pub fn run_from_daemonize(
    options: Arc<Options>,
    monitor_channels: Option<monitor_rpc::Channels>,
    start_cause: server_status::StartCause,
) {
    install_panic_hook();
    let result = std::thread::Builder::new()
        .name("flow_server_main".to_string())
        .stack_size(SERVER_MAIN_THREAD_STACK_SIZE)
        .spawn(move || {
            std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| {
                run(options, monitor_channels, start_cause);
            }))
        })
        .expect("failed to spawn flow_server_main thread")
        .join()
        .unwrap_or_else(Err);
    match result {
        Ok(()) => {}
        Err(e) => {
            let err_msg = panic_message(&*e);
            let err_display: &dyn std::fmt::Display = &err_msg;
            let status = flow_common_exit_status::exit_status_for_panic_message(&err_msg);
            match status {
                FlowExitStatus::OutOfSharedMemory => {
                    let msg = exit_msg_of_exception(err_display, "Out of shared memory");
                    flow_hh_logger::info!("{}", msg);
                }
                FlowExitStatus::HashTableFull => {
                    let msg = exit_msg_of_exception(err_display, "Hash table is full");
                    flow_hh_logger::info!("{}", msg);
                }
                FlowExitStatus::HeapFull => {

View on GitHub (pinned to f88ac94bcf)

Solutions

1. Check the thread budget (ulimit -u, /proc/self/status Threads, container pids.max) and raise it or stop other flow instances
2. Verify memory headroom covers the configured stack size; lower FLOW_STACK_SIZE if it is oversized
3. Restart after freeing resources; spawn failure under load spikes is transient
4. Replace the expect with a retry-then-exit path that reports the io::Error from spawn

Example fix

// before
.spawn(move || { ... })
.expect("failed to spawn flow_server_main thread");

// after
let handle = std::thread::Builder::new()
    .name("flow_server_main".to_string())
    .stack_size(SERVER_MAIN_THREAD_STACK_SIZE)
    .spawn(move || { ... });
match handle {
    Ok(h) => { let _ = h.join(); }
    Err(e) => {
        eprintln!("cannot spawn flow_server_main: {e}; check nproc/memory limits");
        std::process::exit(1);
    }
}

// Verify the host can create one more big-stack thread before launching
let probe = std::thread::Builder::new()
    .stack_size(SERVER_MAIN_THREAD_STACK_SIZE)
    .spawn(Option::<u8>::None);
if probe.is_err() {
    eprintln!("thread budget exhausted; raise ulimit -u / memory before starting flow");
}

When it happens

Trigger: Starting the server when RLIMIT_NPROC / cgroup pids.max is exhausted, when memory limits forbid mapping SERVER_MAIN_THREAD_STACK_SIZE, or when FLOW_STACK_SIZE or the compiled-in stack default is set to an unreasonable value for the host.

Common situations: Language servers in memory/pids-constrained containers; many flow servers per host; containers whose memory limit was lowered after initial sizing; FLOW_STACK_SIZE tuned very large.

Related errors

• failed to spawn init thread
• failed to spawn the command executor
• Failed to create tokio runtime
• failed to spawn connection thread
• failed to spawn recheck_cancel_monitor thread

AI-assisted analysis of facebook/flow@f88ac94bcf (2026-08-20). Data as JSON: /api/errors/59fcfe691afb0bd1. Report an issue: GitHub.