{"record":{"id":"de6a3e2d05038998","repo":"facebook/flow","slug":"failed-to-spawn-glean-runner-visit-timeout-thread","errorCode":null,"errorMessage":"failed to spawn glean_runner_visit_timeout thread","messagePattern":"failed to spawn glean_runner_visit_timeout thread","errorType":"panic","errorClass":null,"httpStatus":null,"severity":"error","filePath":"rust_port/crates/flow_cli/src/glean_runner.rs","lineNumber":2139,"sourceCode":"            let timeout = config.glean_timeout;\n            let file = file.to_absolute();\n            let timeout_thread = std::thread::Builder::new()\n                .name(\"glean_runner_visit_timeout\".to_string())\n                .spawn(move || {\n                    match timeout_receiver\n                        .recv_timeout(std::time::Duration::from_secs(timeout as u64))\n                    {\n                        Ok(()) | Err(std::sync::mpsc::RecvTimeoutError::Disconnected) => {}\n                        Err(std::sync::mpsc::RecvTimeoutError::Timeout) => {\n                            let msg = format!(\"Timed out visiting: {}\", file);\n                            flow_common_exit_status::exit_with_msg(\n                                flow_common_exit_status::FlowExitStatus::UnknownError,\n                                &msg,\n                            );\n                        }\n                    }\n                })\n                .expect(\"failed to spawn glean_runner_visit_timeout thread\");\n            let ret = f();\n            match timeout_canceller.send(()) {\n                Ok(()) | Err(_) => {}\n            }\n            timeout_thread\n                .join()\n                .expect(\"glean_runner_visit_timeout thread panicked\");\n            ret\n        } else {\n            f()\n        }\n    }\n}\n\n#[derive(Clone, Debug)]\nstruct GleanAccumulator {\n    files_analyzed: usize,\n    json_filenames: BTreeSet<String>,","sourceCodeStart":2121,"sourceCodeEnd":2157,"githubUrl":"https://github.com/facebook/flow/blob/f88ac94bcf6992f5d5a158854d94613ebb92c6e6/rust_port/crates/flow_cli/src/glean_runner.rs#L2121-L2157","documentation":"During `flow glean` (indexing into Glean), each file visit that has a timeout configured runs under a watchdog: a thread named glean_runner_visit_timeout waits on a channel and exits the process with 'Timed out visiting' if the deadline passes. This expect fires when std::thread::Builder::spawn cannot create that watchdog thread at all — an OS resource failure (thread/pid limit or memory), not a timeout.","triggerScenarios":"Indexing a large repo with per-file timeout threads while RLIMIT_NPROC / cgroup pids.max is nearly exhausted, or memory is too low to map another thread stack; each visited file spawns a watchdog, multiplying spawn attempts until one fails.","commonSituations":"Restricted CI containers (--pids-limit) running full-repo gleans; machines under heavy thread load; memory pressure from the glean run itself (visit f() is memory-hungry) preventing stack allocation for the watchdog.","solutions":["Raise the process/thread budget (docker --pids-limit, systemd TasksMax, `ulimit -u`) or reduce concurrent load on the machine.","Free memory and retry the glean run.","Run the glean without the per-file timeout so no watchdog thread is spawned.","Maintainer fix: if spawn fails, run f() unguarded (log a warning) or exit with a clear message instead of panicking."],"exampleFix":"// before\nlet timeout_thread = std::thread::Builder::new()\n    .name(\"glean_runner_visit_timeout\".to_string())\n    .spawn(move || { /* recv_timeout watchdog */ })\n    .expect(\"failed to spawn glean_runner_visit_timeout thread\");\nlet ret = f();\n\n// after\nlet timeout_thread = std::thread::Builder::new()\n    .name(\"glean_runner_visit_timeout\".to_string())\n    .spawn(move || { /* recv_timeout watchdog */ });\nif timeout_thread.is_err() {\n    eprintln!(\"warning: could not start visit watchdog; visiting without timeout\");\n    return f();\n}\nlet timeout_thread = timeout_thread.unwrap();\nlet ret = f();","handlingStrategy":"fallback","validationCode":"let threads_budget_ok = std::fs::read_to_string(\"/proc/self/status\")\n    .map(|s| s.lines().find(|l| l.starts_with(\"Threads:\"))\n        .and_then(|l| l.split_whitespace().nth(1).and_then(|n| n.parse::<usize>().ok()))\n    .map(|n| n < 90) // stay well under typical pids.max=100\n    .unwrap_or(true);","typeGuard":null,"tryCatchPattern":"let watchdog = std::thread::Builder::new()\n    .name(\"glean_runner_visit_timeout\".to_string())\n    .spawn(move |/* watchdog closure */|);\nmatch watchdog {\n    Ok(thread) => { /* run f() with watchdog as before */ }\n    Err(e) => {\n        eprintln!(\"warning: cannot start visit watchdog ({}); continuing without timeout\", e);\n        f()\n    }\n}","preventionTips":["Raise pids/thread limits (docker --pids-limit, systemd TasksMax, ulimit -u) before full-repo gleans.","Run gleans on machines with headroom in memory and thread count.","Watch for memory pressure during indexing — the watchdog thread needs its own stack."],"tags":["thread","spawn","resource-limits","glean","timeout","panic"],"backgroundTag":"thread-spawn-failure","analyzedSha":"f88ac94bcf6992f5d5a158854d94613ebb92c6e6","analyzedAt":"2026-08-20T10:41:37.992Z","schemaVersion":2},"datasetVersion":"2026-08-23T08:06:27.607Z"}