facebook/flow · error

failed to spawn glean_runner_visit_timeout thread

Error message

failed to spawn glean_runner_visit_timeout thread

What it means

During `flow glean` (indexing into Glean), each file visit that has a timeout configured runs under a watchdog: a thread named glean_runner_visit_timeout waits on a channel and exits the process with 'Timed out visiting' if the deadline passes. This expect fires when std::thread::Builder::spawn cannot create that watchdog thread at all — an OS resource failure (thread/pid limit or memory), not a timeout.

Source

Thrown at rust_port/crates/flow_cli/src/glean_runner.rs:2139

            let timeout = config.glean_timeout;
            let file = file.to_absolute();
            let timeout_thread = std::thread::Builder::new()
                .name("glean_runner_visit_timeout".to_string())
                .spawn(move || {
                    match timeout_receiver
                        .recv_timeout(std::time::Duration::from_secs(timeout as u64))
                    {
                        Ok(()) | Err(std::sync::mpsc::RecvTimeoutError::Disconnected) => {}
                        Err(std::sync::mpsc::RecvTimeoutError::Timeout) => {
                            let msg = format!("Timed out visiting: {}", file);
                            flow_common_exit_status::exit_with_msg(
                                flow_common_exit_status::FlowExitStatus::UnknownError,
                                &msg,
                            );
                        }
                    }
                })
                .expect("failed to spawn glean_runner_visit_timeout thread");
            let ret = f();
            match timeout_canceller.send(()) {
                Ok(()) | Err(_) => {}
            }
            timeout_thread
                .join()
                .expect("glean_runner_visit_timeout thread panicked");
            ret
        } else {
            f()
        }
    }
}

#[derive(Clone, Debug)]
struct GleanAccumulator {
    files_analyzed: usize,
    json_filenames: BTreeSet<String>,

View on GitHub (pinned to f88ac94bcf)

Solutions

1. Raise the process/thread budget (docker --pids-limit, systemd TasksMax, `ulimit -u`) or reduce concurrent load on the machine.
2. Free memory and retry the glean run.
3. Run the glean without the per-file timeout so no watchdog thread is spawned.
4. Maintainer fix: if spawn fails, run f() unguarded (log a warning) or exit with a clear message instead of panicking.

Example fix

// before
let timeout_thread = std::thread::Builder::new()
    .name("glean_runner_visit_timeout".to_string())
    .spawn(move || { /* recv_timeout watchdog */ })
    .expect("failed to spawn glean_runner_visit_timeout thread");
let ret = f();

// after
let timeout_thread = std::thread::Builder::new()
    .name("glean_runner_visit_timeout".to_string())
    .spawn(move || { /* recv_timeout watchdog */ });
if timeout_thread.is_err() {
    eprintln!("warning: could not start visit watchdog; visiting without timeout");
    return f();
}
let timeout_thread = timeout_thread.unwrap();
let ret = f();

let threads_budget_ok = std::fs::read_to_string("/proc/self/status")
    .map(|s| s.lines().find(|l| l.starts_with("Threads:"))
        .and_then(|l| l.split_whitespace().nth(1).and_then(|n| n.parse::<usize>().ok()))
    .map(|n| n < 90) // stay well under typical pids.max=100
    .unwrap_or(true);

let watchdog = std::thread::Builder::new()
    .name("glean_runner_visit_timeout".to_string())
    .spawn(move |/* watchdog closure */|);
match watchdog {
    Ok(thread) => { /* run f() with watchdog as before */ }
    Err(e) => {
        eprintln!("warning: cannot start visit watchdog ({}); continuing without timeout", e);
        f()
    }
}

When it happens

Trigger: Indexing a large repo with per-file timeout threads while RLIMIT_NPROC / cgroup pids.max is nearly exhausted, or memory is too low to map another thread stack; each visited file spawns a watchdog, multiplying spawn attempts until one fails.

Common situations: Restricted CI containers (--pids-limit) running full-repo gleans; machines under heavy thread load; memory pressure from the glean run itself (visit f() is memory-hungry) preventing stack allocation for the watchdog.

Understand the failure class

• Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

• failed to spawn connect_and_make_request_timed thread
• failed to spawn flow_cli_main thread
• Daemon::spawn: timed out waiting for child to connect
• Failed to create tokio runtime
• failed to spawn flow_server_main thread

AI-assisted analysis of facebook/flow@f88ac94bcf (2026-08-20). Data as JSON: /api/errors/de6a3e2d05038998. Report an issue: GitHub.