facebook/flow · critical

failed to spawn the command executor

Error message

failed to spawn the command executor

What it means

ServerOrchestrator's CommandExecutor::start spawns the flow-command thread (stack size DEFAULT_STACK_SIZE, overridable via FLOW_STACK_SIZE) and expects success (rust_port/crates/flow_server_env/src/server_orchestrator.rs:168-174); a second expect then blocks on the started signal. A spawn failure means the OS could not create the command-executor thread, so orchestrator startup — and the server with it — aborts. Panics inside the thread are handled by run_with_panic_handler; this expect covers only thread creation.

Source

Thrown at rust_port/crates/flow_server_env/src/server_orchestrator.rs:174

    }
}

impl Default for ServerOrchestrator {
    fn default() -> Self {
        Self::new()
    }
}

impl CommandExecutor {
    fn start(self, env: EnvRef) -> RunningServerOrchestrator {
        let (started, wait_for_started) = std::sync::mpsc::channel();
        let control = self.control.clone();
        let builder = std::thread::Builder::new().name("flow-command".to_string());
        #[cfg(not(target_arch = "wasm32"))]
        let builder = builder.stack_size(flow_utils_concurrency::thread_pool::DEFAULT_STACK_SIZE);
        let thread = builder
            .spawn(move || self.run_with_panic_handler(env, started))
            .expect("failed to spawn the command executor");
        wait_for_started
            .recv()
            .expect("the command executor should start before the server publishes readiness");
        RunningServerOrchestrator {
            control,
            thread: Some(thread),
        }
    }

    fn run_with_panic_handler(self, env: EnvRef, started: std::sync::mpsc::Sender<()>) {
        if let Err(payload) = std::panic::catch_unwind(AssertUnwindSafe(|| self.run(env, started)))
        {
            let message = payload
                .downcast_ref::<&str>()
                .copied()
                .or_else(|| payload.downcast_ref::<String>().map(String::as_str))
                .unwrap_or("unknown panic");
            flow_hh_logger::error!("Unhandled exception on the command executor: {}", message);

View on GitHub (pinned to f88ac94bcf)

Solutions

1. Raise ulimit -u / pids.max / memory limits; sanity-check FLOW_STACK_SIZE against available memory
2. Lower --max-workers so startup leaves headroom for orchestrator threads
3. Propagate the spawn io::Error to the caller so the monitor can report a clear startup failure instead of a panic
4. If the sibling expect fired instead ('should start before the server publishes readiness'), the thread spawned but panicked early — inspect the panic-handler logs

Example fix

// before
let thread = builder
    .spawn(move || self.run_with_panic_handler(env, started))
    .expect("failed to spawn the command executor");

// after
let thread = builder
    .spawn(move || self.run_with_panic_handler(env, started))
    .map_err(|e| format!("cannot spawn command executor: {e}; check nproc/memory/FLOW_STACK_SIZE"))?;

// Validate stack config before server start
if let Ok(s) = std::env::var("FLOW_STACK_SIZE") {
    let bytes: usize = s.parse().expect("FLOW_STACK_SIZE must be a number");
    let workers = options.max_workers.max(1) as usize;
    assert!(bytes.saturating_mul(workers) < available_memory(), "thread stacks exceed memory");
}

When it happens

Trigger: Server startup at the thread or memory ceiling: worker pool plus connection threads already live, FLOW_STACK_SIZE set so large the new thread's stack cannot be mapped, or cgroup pids.max reached at exactly this spawn.

Common situations: High --max-workers on small instances; FLOW_STACK_SIZE raised for deeply recursive inputs without raising memory limits; nested or embedded servers sharing one cgroup's thread budget.

Related errors

• failed to spawn flow_server_main thread
• failed to spawn init thread
• Failed to create tokio runtime
• failed to spawn connection thread
• failed to spawn recheck_cancel_monitor thread

AI-assisted analysis of facebook/flow@f88ac94bcf (2026-08-20). Data as JSON: /api/errors/63fa1367c0ca1ca7. Report an issue: GitHub.