facebook/flow · critical

failed to spawn connection thread

Error message

failed to spawn connection thread

What it means

For every accepted client connection the standalone server spawns a handler thread with CONNECTION_THREAD_STACK_SIZE (2 MiB) and expects spawn to succeed (rust_port/crates/flow_server/src/standalone.rs:416). Connections are capped by ConnectionSlots/MAX_CONNECTION_THREADS (128), but each client still costs one OS thread; when the OS refuses to create it the expect panics on the accept thread, killing the whole server process for all clients.

Source

Thrown at rust_port/crates/flow_server/src/standalone.rs:416

                    let socket_path = socket_path.clone();
                    let orchestrator = orchestrator.clone();
                    std::thread::Builder::new()
                        .stack_size(CONNECTION_THREAD_STACK_SIZE)
                        .spawn(move || {
                            let _slot_guard = slot_guard;
                            handle_connection(
                                &state,
                                &options,
                                &committed_heap,
                                &orchestrator,
                                pool_workers,
                                stream,
                                &pids_path,
                                &lock_path,
                                &socket_path,
                            );
                        })
                        .expect("failed to spawn connection thread");
                }
                Err(e) => {
                    eprintln!("Error accepting connection: {}", e);
                }
            }
        }
    }
}

enum RecheckOutcome {
    Ok,
}

enum CommandAttempt<T> {
    Completed(T),
    RetryAfterRecheck,
}

View on GitHub (pinned to f88ac94bcf)

Solutions

1. Raise the host thread/memory budget (ulimit -u, pids.max, memory limit) to cover MAX_CONNECTION_THREADS (128) x 2 MiB stacks plus workers
2. Reduce concurrent clients or lower MAX_CONNECTION_THREADS if you control the build
3. Replace the expect with a graceful path: log the spawn error, drop the stream, keep the accept loop alive
4. Monitor /proc/<pid>/status Threads and restart the server before it approaches the ceiling

Example fix

// before
.expect("failed to spawn connection thread");

// after: per-connection failure does not kill the accept loop
if let Err(e) = builder.spawn(move || handle_connection(/* ... */)) {
    eprintln!("Error spawning connection thread: {e}; dropping client");
    drop(stream);
}

// Refuse new clients before the thread budget runs out
let threads = current_thread_count(); // read /proc/self/status Threads
if threads + 1 >= max_thread_budget() {
    eprintln!("Refusing connection: thread budget nearly exhausted");
    drop(stream);
    continue;
}

When it happens

Trigger: A burst of concurrent clients (up to 128) crossing the host's thread or memory limit exactly when a new connection thread is spawned; cgroup pids.max exhausted by worker pool plus existing connection threads; low-memory environments where 128 x 2 MiB stacks cannot be mapped.

Common situations: Editor/LSP load spikes opening many connections; containers with low memory or pids limits; long-lived servers leaking threads elsewhere in the process.

Related errors

• failed to spawn flow_server_main thread
• failed to spawn init thread
• failed to spawn recheck_cancel_monitor thread
• failed to spawn wait_for_anything thread
• failed to spawn the command executor

AI-assisted analysis of facebook/flow@f88ac94bcf (2026-08-20). Data as JSON: /api/errors/b1b69b1345a1f0ba. Report an issue: GitHub.