facebook/flow · error

failed to spawn recheck_cancel_monitor thread

Error message

failed to spawn recheck_cancel_monitor thread

What it means

Rechecks run with a small helper: a recheck_cancel_monitor thread listens on crossbeam channels and stops workers when a newer force-recheck or file-watcher update arrives (rust_port/crates/flow_server/src/standalone.rs:528-543). Spawning that thread is expected to succeed; failure panics mid-setup of a recheck. As with all std::thread spawn failures the root cause is OS-level: thread/process or memory limits at the moment the monitor thread is created.

Source

Thrown at rust_port/crates/flow_server/src/standalone.rs:541

/// This is the Rust port equivalent of OCaml's `cancel_thread` half of
/// `Lwt.pick` inside `run_but_cancel_on_file_changes` (rechecker.ml:228).
fn spawn_recheck_cancel_monitor() -> RecheckCancelMonitor {
    let push_rx = server_monitor_listener_state::subscribe_recheck_pushes();
    let (stop_tx, stop_rx) = crossbeam::channel::bounded::<()>(1);
    let thread = std::thread::Builder::new()
        .name("recheck_cancel_monitor".to_string())
        .spawn(move || {
            crossbeam::channel::select! {
                recv(push_rx) -> _ => {
                    eprintln!(
                        "Canceling recheck because a new force-recheck or file-watcher update arrived"
                    );
                    worker_cancel::stop_workers();
                }
                recv(stop_rx) -> _ => {}
            }
        })
        .expect("failed to spawn recheck_cancel_monitor thread");
    RecheckCancelMonitor {
        stop_tx,
        thread: Some(thread),
    }
}

fn do_rechecks(
    state: &Arc<(Mutex<ServerState>, Condvar)>,
    options: &Arc<Options>,
    committed_heap: &Arc<CommittedHeap>,
    orchestrator: &server_orchestrator::ServerOrchestratorHandle,
    pool_workers: usize,
) -> RecheckOutcome {
    let pool = Arc::new(ThreadPool::with_thread_count(
        flow_utils_concurrency::thread_pool::ThreadCount::NumThreads(
            std::num::NonZeroUsize::new(pool_workers).expect("pool_workers should be positive"),
        ),
    ));

View on GitHub (pinned to f88ac94bcf)

Solutions

1. Increase thread/process limits for the flow server process (ulimit -u, pids.max, UserTasksMax)
2. Lower --max-workers so recheck-time helper threads fit within the budget
3. Make the spawn non-fatal: fall back to running the cancel logic inline or retry the spawn after a delay instead of expecting
4. Check for thread leaks in /proc/<pid>/status before blaming this specific spawn

Example fix

// before
.expect("failed to spawn recheck_cancel_monitor thread");

// after: degrade gracefully when the OS refuses the thread
let thread = match builder.spawn(move || { /* channel select loop */ }) {
    Ok(t) => Some(t),
    Err(e) => {
        eprintln!("Warning: cannot spawn recheck_cancel_monitor: {e}; cancel-on-new-update disabled");
        None
    }
};

let threads = current_thread_count();
if threads >= thread_soft_limit() {
    // skip spawning the cancel monitor; recheck still runs without cancel-on-new-update
    return RecheckCancelMonitor::disabled();
}

When it happens

Trigger: A file-watcher update or forced recheck beginning while the process sits at its thread budget (worker pool plus connection threads already at cgroup pids.max or RLIMIT_NPROC), so even one small monitor thread cannot be created.

Common situations: Heavy recheck activity in constrained containers; systemd UserTasksMax or RLIMIT_NPROC set tight; test harnesses running many server instances per machine.

Related errors

• failed to spawn wait_for_anything thread
• failed to spawn flow_server_main thread
• failed to spawn init thread
• failed to spawn connection thread
• failed to spawn the command executor

AI-assisted analysis of facebook/flow@f88ac94bcf (2026-08-20). Data as JSON: /api/errors/450fc027eefc978f. Report an issue: GitHub.