{"record":{"id":"b1b69b1345a1f0ba","repo":"facebook/flow","slug":"failed-to-spawn-connection-thread","errorCode":null,"errorMessage":"failed to spawn connection thread","messagePattern":"failed to spawn connection thread","errorType":"panic","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"rust_port/crates/flow_server/src/standalone.rs","lineNumber":416,"sourceCode":"                    let socket_path = socket_path.clone();\n                    let orchestrator = orchestrator.clone();\n                    std::thread::Builder::new()\n                        .stack_size(CONNECTION_THREAD_STACK_SIZE)\n                        .spawn(move || {\n                            let _slot_guard = slot_guard;\n                            handle_connection(\n                                &state,\n                                &options,\n                                &committed_heap,\n                                &orchestrator,\n                                pool_workers,\n                                stream,\n                                &pids_path,\n                                &lock_path,\n                                &socket_path,\n                            );\n                        })\n                        .expect(\"failed to spawn connection thread\");\n                }\n                Err(e) => {\n                    eprintln!(\"Error accepting connection: {}\", e);\n                }\n            }\n        }\n    }\n}\n\nenum RecheckOutcome {\n    Ok,\n}\n\nenum CommandAttempt<T> {\n    Completed(T),\n    RetryAfterRecheck,\n}\n","sourceCodeStart":398,"sourceCodeEnd":434,"githubUrl":"https://github.com/facebook/flow/blob/f88ac94bcf6992f5d5a158854d94613ebb92c6e6/rust_port/crates/flow_server/src/standalone.rs#L398-L434","documentation":"For every accepted client connection the standalone server spawns a handler thread with CONNECTION_THREAD_STACK_SIZE (2 MiB) and expects spawn to succeed (rust_port/crates/flow_server/src/standalone.rs:416). Connections are capped by ConnectionSlots/MAX_CONNECTION_THREADS (128), but each client still costs one OS thread; when the OS refuses to create it the expect panics on the accept thread, killing the whole server process for all clients.","triggerScenarios":"A burst of concurrent clients (up to 128) crossing the host's thread or memory limit exactly when a new connection thread is spawned; cgroup pids.max exhausted by worker pool plus existing connection threads; low-memory environments where 128 x 2 MiB stacks cannot be mapped.","commonSituations":"Editor/LSP load spikes opening many connections; containers with low memory or pids limits; long-lived servers leaking threads elsewhere in the process.","solutions":["Raise the host thread/memory budget (ulimit -u, pids.max, memory limit) to cover MAX_CONNECTION_THREADS (128) x 2 MiB stacks plus workers","Reduce concurrent clients or lower MAX_CONNECTION_THREADS if you control the build","Replace the expect with a graceful path: log the spawn error, drop the stream, keep the accept loop alive","Monitor /proc/<pid>/status Threads and restart the server before it approaches the ceiling"],"exampleFix":"// before\n.expect(\"failed to spawn connection thread\");\n\n// after: per-connection failure does not kill the accept loop\nif let Err(e) = builder.spawn(move || handle_connection(/* ... */)) {\n    eprintln!(\"Error spawning connection thread: {e}; dropping client\");\n    drop(stream);\n}","handlingStrategy":"validation","validationCode":"// Refuse new clients before the thread budget runs out\nlet threads = current_thread_count(); // read /proc/self/status Threads\nif threads + 1 >= max_thread_budget() {\n    eprintln!(\"Refusing connection: thread budget nearly exhausted\");\n    drop(stream);\n    continue;\n}","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Size container memory and pids limits for MAX_CONNECTION_THREADS (128) x 2 MiB stacks plus workers","Keep an accept-loop guard on thread count, not just connection count","Load-test concurrent client bursts before deploying"],"tags":["thread-spawn","connection-handling","resource-limits","server-crash","panic"],"backgroundTag":"thread-spawn-failed","analyzedSha":"f88ac94bcf6992f5d5a158854d94613ebb92c6e6","analyzedAt":"2026-08-20T10:41:37.992Z","schemaVersion":2},"datasetVersion":"2026-08-23T08:06:27.607Z"}