{"record":{"id":"e09f3018ee8e8273","repo":"quickwit-oss/quickwit","slug":"failed-to-spawn-thread-pool","errorCode":null,"errorMessage":"failed to spawn thread pool","messagePattern":"failed to spawn thread pool","errorType":"panic","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"quickwit/quickwit-common/src/thread_pool/simple.rs","lineNumber":47,"sourceCode":"struct SimpleThreadPool {\n    thread_pool: Arc<rayon::ThreadPool>,\n    ongoing_tasks: Gauge,\n    pending_tasks: Gauge,\n}\n\nimpl SimpleThreadPool {\n    fn new(name: &'static str, num_threads_opt: Option<usize>) -> SimpleThreadPool {\n        let mut rayon_pool_builder = rayon::ThreadPoolBuilder::new()\n            .thread_name(move |thread_id| format!(\"quickwit-{name}-{thread_id}\"))\n            .panic_handler(move |_my_panic| {\n                error!(\"task running in the quickwit {name} thread pool panicked\");\n            });\n        if let Some(num_threads) = num_threads_opt {\n            rayon_pool_builder = rayon_pool_builder.num_threads(num_threads);\n        }\n        let thread_pool = rayon_pool_builder\n            .build()\n            .expect(\"failed to spawn thread pool\");\n        let labels = labels!(\"pool\" => name);\n        let ongoing_tasks = gauge!(parent: THREAD_POOL_ONGOING_TASKS, labels: [labels]);\n        let pending_tasks = gauge!(parent: THREAD_POOL_PENDING_TASKS, labels: [labels]);\n        SimpleThreadPool {\n            thread_pool: Arc::new(thread_pool),\n            ongoing_tasks,\n            pending_tasks,\n        }\n    }\n\n    /// Function similar to `tokio::spawn_blocking`.\n    ///\n    /// Here are two important differences however:\n    ///\n    /// 1) The task runs on a rayon thread pool managed by Quickwit. This pool is specifically used\n    ///    only to run CPU-intensive work and is configured to contain `num_cpus` cores.\n    ///\n    /// 2) Before the task is effectively scheduled, we check that the spawner is still interested","sourceCodeStart":29,"sourceCodeEnd":65,"githubUrl":"https://github.com/quickwit-oss/quickwit/blob/a39730c5cdcd1a4fe798403737ae293999ea21f8/quickwit/quickwit-common/src/thread_pool/simple.rs#L29-L65","documentation":"This `.expect(\"failed to spawn thread pool\")` panic occurs in `SimpleThreadPool::new` (quickwit-common/src/thread_pool/simple.rs) when the underlying rayon `ThreadPoolBuilder::build()` returns `None`. Rayon fails to build a pool when it cannot spawn the requested worker threads (or the requested thread count is 0), typically due to operating-system resource limits. Since a thread pool with no workers is unusable, the constructor aborts by panicking.","triggerScenarios":"Constructing a `SimpleThreadPool` via `new(name, num_threads, ...)` when rayon cannot spawn the requested threads: process/thread limit (`RLIMIT_NPROC`, `ulimit -u`, cgroup pids.max, container thread limits) reached, memory exhaustion at thread creation, or configuring `num_threads` such that rayon refuses it.","commonSituations":"Running Quickwit in tightly constrained Docker/Kubernetes pods with low `pids` limits or low `RLIMIT_NPROC`; heavily loaded hosts that already exhaust the thread budget; misconfigured pool sizing in `quickwit.yaml` requesting an impossible thread count.","solutions":["Raise the OS thread limits: increase `ulimit -u` (RLIMIT_NPROC) or the container/Kubernetes `pids.max` limit and restart.","Reduce the configured `num_threads` for the pool (or unset it to use num_cpus) so thread spawning succeeds within current limits.","Check host memory/pressure: thread creation can fail under memory exhaustion; free memory or add headroom.","If running many pools, consolidate pools or lower total worker counts so the cumulative thread count fits the limit."],"exampleFix":"// before (config asking for too many threads under a pids limit)\n[searcher]\nconcurrency = 1024\n// after: lower concurrency / let it default\n[searcher]\nconcurrency = 8\n# plus, at OS level:\n# ulimit -u 4096  (or raise the pod pids limit)","handlingStrategy":"validation","validationCode":"// Check thread headroom before constructing the pool (Linux)\nuse std::fs;\nlet tasks = fs::read_to_string(\"/proc/self/status\")\n    .ok()\n    .and_then(|s| s.lines().find(|l| l.starts_with(\"Threads:\"))?.split_whitespace().last()?.parse::<u64>().ok())\n    .unwrap_or(0);\nlet limit = nix::sys::resource::getrlimit(nix::sys::resource::Resource::RLIMIT_NPROC)\n    .ok().map(|(soft, _)| soft).unwrap_or(u64::MAX);\nassert!(tasks + requested_threads < limit, \"insufficient thread headroom\");","typeGuard":"fn can_spawn_threads(requested: usize) -> bool {\n    // heuristic: current process threads + requested below rlimit\n    fs::read_to_string(\"/proc/self/status\").map(|s| {\n        let cur: u64 = s.lines().find_map(|l| l.strip_prefix(\"Threads:\")?.trim().parse().ok()).unwrap_or(0);\n        let (soft, _) = nix::sys::resource::getrlimit(nix::sys::resource::Resource::RLIMIT_NPROC).ok()?;\n        cur + requested as u64 < soft\n    }).unwrap_or(true)\n}","tryCatchPattern":"// Constructor panics; wrap pool creation at startup\nlet pool = std::panic::catch_unwind(|| SimpleThreadPool::new(\"search\", n, throttle))\n    .map_err(|_| anyhow::anyhow!(\"failed to spawn thread pool; check ulimit/pids limit\"))?;","preventionTips":["Set generous RLIMIT_NPROC / container pids limits before deploying.","Avoid hard-coding large num_threads values; let pools default to num_cpus.","Monitor process thread counts in production.","Validate pool configuration early at startup, not lazily."],"tags":["rust","rayon","thread-pool","resource-limit","startup"],"backgroundTag":"module-init-failed","analyzedSha":"a39730c5cdcd1a4fe798403737ae293999ea21f8","analyzedAt":"2026-09-08T13:19:37.784Z","contentChangedAt":"2026-09-08T13:19:37.784Z","schemaVersion":2},"datasetVersion":"2026-09-14T11:17:12.474Z"}