facebook/flow · error

failed to spawn connect_and_make_request_timed thread

Error message

failed to spawn connect_and_make_request_timed thread

What it means

For commands that enforce a timeout, the client spawns a worker thread (named connect_and_make_request_timed) that performs connect_and_make_request_inner and sends the result over an mpsc channel; recv_timeout on the channel enforces the deadline. This expect fires when std::thread::Builder::spawn itself returns Err, i.e. the OS refused to create a thread (pid/thread limit, memory for the thread stack). Nothing about the flow server is involved yet — the panic happens before any request starts.

Source

Thrown at rust_port/crates/flow_cli/src/command_utils.rs:3276

            let root_owned = root.to_path_buf();
            let request_clone = request.clone();
            let initial_retries = connect_flags.retries;
            std::thread::Builder::new()
                .name("connect_and_make_request_timed".to_string())
                .spawn(move || {
                    let response = connect_and_make_request_inner(
                        &flowconfig_name_owned,
                        &connect_flags_clone,
                        &root_owned,
                        &request_clone,
                        initial_retries,
                    );
                    match tx.send(response) {
                        Ok(()) => {}
                        Err(_) => {}
                    }
                })
                .expect("failed to spawn connect_and_make_request_timed thread");
            match rx.recv_timeout(std::time::Duration::from_secs(timeout as u64)) {
                Ok(response) => response,
                Err(std::sync::mpsc::RecvTimeoutError::Timeout) => flow_common_exit::exit(
                    flow_common_exit::FlowExitStatus::OutOfTime,
                    Some("Timeout exceeded, exiting"),
                ),
                Err(std::sync::mpsc::RecvTimeoutError::Disconnected) => flow_common_exit::exit(
                    flow_common_exit::FlowExitStatus::UnknownError,
                    Some("Inner connect thread panicked, exiting"),
                ),
            }
        }
    }
}

pub(crate) fn failwith_bad_response(
    request: &server_prot::request::Command,
    response: &server_prot::response::Response,

View on GitHub (pinned to f88ac94bcf)

Solutions

1. Raise the process/thread budget: increase docker --pids-limit / systemd TasksMax / `ulimit -u`, or reduce the number of concurrent processes on the machine.
2. Free memory so the thread stack can be allocated; retry once load drops.
3. Run the command without the timeout option so the timed code path is avoided (if the command allows it).
4. Maintainer fix: on spawn failure, fall back to calling connect_and_make_request_inner inline (losing the timeout guard) or exit with a clear diagnostic instead of panicking.

Example fix

// before
let handle = std::thread::Builder::new()
    .name("connect_and_make_request_timed".to_string())
    .spawn(move || { /* inner request, tx.send */ })
    .expect("failed to spawn connect_and_make_request_timed thread");

// after
let spawned = std::thread::Builder::new()
    .name("connect_and_make_request_timed".to_string())
    .spawn(move || { /* inner request, tx.send */ });
let handle = match spawned {
    Ok(handle) => handle,
    Err(e) => {
        eprintln!("could not start timeout worker ({}); running without timeout", e);
        return connect_and_make_request_inner(/* ... */);
    }
};

let can_spawn_thread = std::thread::available_parallelism().is_ok(); // rough capacity signal only
let pid_budget_ok = std::fs::read_to_string("/proc/self/status")
    .map(|s| s.lines().any(|l| l.starts_with("Threads:") && l.split_whitespace().nth(1).map(|n| n.parse::<usize>().map(|n| n < 100).unwrap_or(true)).unwrap_or(true)))
    .unwrap_or(true);

let spawned = std::thread::Builder::new()
    .name("connect_and_make_request_timed".to_string())
    .spawn(move || { /* ... */ });
match spawned {
    Ok(handle) => { /* wait on rx.recv_timeout as before */ }
    Err(e) => {
        eprintln!("warning: no threads available ({}); running without timeout", e);
        connect_and_make_request_inner(/* ... */)
    }
}

When it happens

Trigger: RLIMIT_NPROC / `ulimit -u` exhausted, a container cgroup pids.max reached, insufficient memory to map the default thread stack, or a sandbox thread cap — at the moment a timeout-flagged flow command runs.

Common situations: CI containers started with docker `--pids-limit` (e.g. 100) that already run many processes/threads; systemd TasksMax reached; heavy parallel test matrices saturating thread counts; memory pressure preventing stack allocation.

Related errors

• failed to spawn glean_runner_visit_timeout thread
• failed to spawn flow_cli_main thread
• Daemon::spawn: timed out waiting for child to connect
• Failed to create tokio runtime
• failed to spawn flow_server_main thread

AI-assisted analysis of facebook/flow@f88ac94bcf (2026-08-20). Data as JSON: /api/errors/4bb3131f40333751. Report an issue: GitHub.