{"record":{"id":"63fa1367c0ca1ca7","repo":"facebook/flow","slug":"failed-to-spawn-the-command-executor","errorCode":null,"errorMessage":"failed to spawn the command executor","messagePattern":"failed to spawn the command executor","errorType":"panic","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"rust_port/crates/flow_server_env/src/server_orchestrator.rs","lineNumber":174,"sourceCode":"    }\n}\n\nimpl Default for ServerOrchestrator {\n    fn default() -> Self {\n        Self::new()\n    }\n}\n\nimpl CommandExecutor {\n    fn start(self, env: EnvRef) -> RunningServerOrchestrator {\n        let (started, wait_for_started) = std::sync::mpsc::channel();\n        let control = self.control.clone();\n        let builder = std::thread::Builder::new().name(\"flow-command\".to_string());\n        #[cfg(not(target_arch = \"wasm32\"))]\n        let builder = builder.stack_size(flow_utils_concurrency::thread_pool::DEFAULT_STACK_SIZE);\n        let thread = builder\n            .spawn(move || self.run_with_panic_handler(env, started))\n            .expect(\"failed to spawn the command executor\");\n        wait_for_started\n            .recv()\n            .expect(\"the command executor should start before the server publishes readiness\");\n        RunningServerOrchestrator {\n            control,\n            thread: Some(thread),\n        }\n    }\n\n    fn run_with_panic_handler(self, env: EnvRef, started: std::sync::mpsc::Sender<()>) {\n        if let Err(payload) = std::panic::catch_unwind(AssertUnwindSafe(|| self.run(env, started)))\n        {\n            let message = payload\n                .downcast_ref::<&str>()\n                .copied()\n                .or_else(|| payload.downcast_ref::<String>().map(String::as_str))\n                .unwrap_or(\"unknown panic\");\n            flow_hh_logger::error!(\"Unhandled exception on the command executor: {}\", message);","sourceCodeStart":156,"sourceCodeEnd":192,"githubUrl":"https://github.com/facebook/flow/blob/f88ac94bcf6992f5d5a158854d94613ebb92c6e6/rust_port/crates/flow_server_env/src/server_orchestrator.rs#L156-L192","documentation":"ServerOrchestrator's CommandExecutor::start spawns the flow-command thread (stack size DEFAULT_STACK_SIZE, overridable via FLOW_STACK_SIZE) and expects success (rust_port/crates/flow_server_env/src/server_orchestrator.rs:168-174); a second expect then blocks on the started signal. A spawn failure means the OS could not create the command-executor thread, so orchestrator startup — and the server with it — aborts. Panics inside the thread are handled by run_with_panic_handler; this expect covers only thread creation.","triggerScenarios":"Server startup at the thread or memory ceiling: worker pool plus connection threads already live, FLOW_STACK_SIZE set so large the new thread's stack cannot be mapped, or cgroup pids.max reached at exactly this spawn.","commonSituations":"High --max-workers on small instances; FLOW_STACK_SIZE raised for deeply recursive inputs without raising memory limits; nested or embedded servers sharing one cgroup's thread budget.","solutions":["Raise ulimit -u / pids.max / memory limits; sanity-check FLOW_STACK_SIZE against available memory","Lower --max-workers so startup leaves headroom for orchestrator threads","Propagate the spawn io::Error to the caller so the monitor can report a clear startup failure instead of a panic","If the sibling expect fired instead ('should start before the server publishes readiness'), the thread spawned but panicked early — inspect the panic-handler logs"],"exampleFix":"// before\nlet thread = builder\n    .spawn(move || self.run_with_panic_handler(env, started))\n    .expect(\"failed to spawn the command executor\");\n\n// after\nlet thread = builder\n    .spawn(move || self.run_with_panic_handler(env, started))\n    .map_err(|e| format!(\"cannot spawn command executor: {e}; check nproc/memory/FLOW_STACK_SIZE\"))?;","handlingStrategy":"validation","validationCode":"// Validate stack config before server start\nif let Ok(s) = std::env::var(\"FLOW_STACK_SIZE\") {\n    let bytes: usize = s.parse().expect(\"FLOW_STACK_SIZE must be a number\");\n    let workers = options.max_workers.max(1) as usize;\n    assert!(bytes.saturating_mul(workers) < available_memory(), \"thread stacks exceed memory\");\n}","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Keep FLOW_STACK_SIZE proportional to available memory divided by worker count","Leave thread headroom at startup for orchestrator threads","Surface spawn io::Errors in logs so limits problems are diagnosable"],"tags":["thread-spawn","orchestrator","server-startup","resource-limits","panic"],"backgroundTag":"thread-spawn-failed","analyzedSha":"f88ac94bcf6992f5d5a158854d94613ebb92c6e6","analyzedAt":"2026-08-20T10:41:37.992Z","schemaVersion":2},"datasetVersion":"2026-08-23T08:06:27.607Z"}