{"record":{"id":"450fc027eefc978f","repo":"facebook/flow","slug":"failed-to-spawn-recheck-cancel-monitor-thread","errorCode":null,"errorMessage":"failed to spawn recheck_cancel_monitor thread","messagePattern":"failed to spawn recheck_cancel_monitor thread","errorType":"panic","errorClass":null,"httpStatus":null,"severity":"error","filePath":"rust_port/crates/flow_server/src/standalone.rs","lineNumber":541,"sourceCode":"/// This is the Rust port equivalent of OCaml's `cancel_thread` half of\n/// `Lwt.pick` inside `run_but_cancel_on_file_changes` (rechecker.ml:228).\nfn spawn_recheck_cancel_monitor() -> RecheckCancelMonitor {\n    let push_rx = server_monitor_listener_state::subscribe_recheck_pushes();\n    let (stop_tx, stop_rx) = crossbeam::channel::bounded::<()>(1);\n    let thread = std::thread::Builder::new()\n        .name(\"recheck_cancel_monitor\".to_string())\n        .spawn(move || {\n            crossbeam::channel::select! {\n                recv(push_rx) -> _ => {\n                    eprintln!(\n                        \"Canceling recheck because a new force-recheck or file-watcher update arrived\"\n                    );\n                    worker_cancel::stop_workers();\n                }\n                recv(stop_rx) -> _ => {}\n            }\n        })\n        .expect(\"failed to spawn recheck_cancel_monitor thread\");\n    RecheckCancelMonitor {\n        stop_tx,\n        thread: Some(thread),\n    }\n}\n\nfn do_rechecks(\n    state: &Arc<(Mutex<ServerState>, Condvar)>,\n    options: &Arc<Options>,\n    committed_heap: &Arc<CommittedHeap>,\n    orchestrator: &server_orchestrator::ServerOrchestratorHandle,\n    pool_workers: usize,\n) -> RecheckOutcome {\n    let pool = Arc::new(ThreadPool::with_thread_count(\n        flow_utils_concurrency::thread_pool::ThreadCount::NumThreads(\n            std::num::NonZeroUsize::new(pool_workers).expect(\"pool_workers should be positive\"),\n        ),\n    ));","sourceCodeStart":523,"sourceCodeEnd":559,"githubUrl":"https://github.com/facebook/flow/blob/f88ac94bcf6992f5d5a158854d94613ebb92c6e6/rust_port/crates/flow_server/src/standalone.rs#L523-L559","documentation":"Rechecks run with a small helper: a recheck_cancel_monitor thread listens on crossbeam channels and stops workers when a newer force-recheck or file-watcher update arrives (rust_port/crates/flow_server/src/standalone.rs:528-543). Spawning that thread is expected to succeed; failure panics mid-setup of a recheck. As with all std::thread spawn failures the root cause is OS-level: thread/process or memory limits at the moment the monitor thread is created.","triggerScenarios":"A file-watcher update or forced recheck beginning while the process sits at its thread budget (worker pool plus connection threads already at cgroup pids.max or RLIMIT_NPROC), so even one small monitor thread cannot be created.","commonSituations":"Heavy recheck activity in constrained containers; systemd UserTasksMax or RLIMIT_NPROC set tight; test harnesses running many server instances per machine.","solutions":["Increase thread/process limits for the flow server process (ulimit -u, pids.max, UserTasksMax)","Lower --max-workers so recheck-time helper threads fit within the budget","Make the spawn non-fatal: fall back to running the cancel logic inline or retry the spawn after a delay instead of expecting","Check for thread leaks in /proc/<pid>/status before blaming this specific spawn"],"exampleFix":"// before\n.expect(\"failed to spawn recheck_cancel_monitor thread\");\n\n// after: degrade gracefully when the OS refuses the thread\nlet thread = match builder.spawn(move || { /* channel select loop */ }) {\n    Ok(t) => Some(t),\n    Err(e) => {\n        eprintln!(\"Warning: cannot spawn recheck_cancel_monitor: {e}; cancel-on-new-update disabled\");\n        None\n    }\n};","handlingStrategy":"validation","validationCode":"let threads = current_thread_count();\nif threads >= thread_soft_limit() {\n    // skip spawning the cancel monitor; recheck still runs without cancel-on-new-update\n    return RecheckCancelMonitor::disabled();\n}","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Reserve headroom for helper threads when sizing --max-workers","Raise cgroup pids.max / ulimit -u on hosts running recheck-heavy workloads","Track thread count trends to detect leaks before a recheck triggers the crash"],"tags":["thread-spawn","recheck","file-watcher","resource-limits","panic"],"backgroundTag":"thread-spawn-failed","analyzedSha":"f88ac94bcf6992f5d5a158854d94613ebb92c6e6","analyzedAt":"2026-08-20T10:41:37.992Z","schemaVersion":2},"datasetVersion":"2026-08-23T08:06:27.607Z"}