{"record":{"id":"77f281bf7ae91710","repo":"quickwit-oss/quickwit","slug":"failed-to-spawn-thread-pool-77f281","errorCode":null,"errorMessage":"failed to spawn thread pool","messagePattern":"failed to spawn thread pool","errorType":"panic","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"quickwit/quickwit-common/src/thread_pool/with_priority.rs","lineNumber":141,"sourceCode":"    /// The default priority.\n    Normal,\n    /// A high-priority task is scheduled before normal-priority tasks that are still pending.\n    High,\n}\n\nimpl ThreadPoolWithPriority {\n    pub fn new(name: &'static str, num_threads_opt: Option<usize>) -> ThreadPoolWithPriority {\n        let mut rayon_pool_builder = rayon::ThreadPoolBuilder::new()\n            .thread_name(move |thread_id| format!(\"quickwit-{name}-{thread_id}\"))\n            .panic_handler(move |_my_panic| {\n                error!(\"task running in the quickwit {name} thread pool panicked\");\n            });\n        if let Some(num_threads) = num_threads_opt {\n            rayon_pool_builder = rayon_pool_builder.num_threads(num_threads);\n        }\n        let thread_pool = rayon_pool_builder\n            .build()\n            .expect(\"failed to spawn thread pool\");\n        let max_running_tasks = thread_pool.current_num_threads();\n        let labels = labels!(\"pool\" => name);\n        let ongoing_tasks = gauge!(parent: THREAD_POOL_ONGOING_TASKS, labels: [labels]);\n        let pending_tasks = gauge!(parent: THREAD_POOL_PENDING_TASKS, labels: [labels]);\n        ThreadPoolWithPriority {\n            inner: Arc::new(ThreadPoolInner {\n                thread_pool: Arc::new(thread_pool),\n                max_running_tasks,\n                num_running_tasks: AtomicUsize::new(0),\n                state: Mutex::new(State {\n                    high_priority_tasks: VecDeque::new(),\n                    normal_priority_tasks: VecDeque::new(),\n                }),\n                ongoing_tasks,\n                pending_tasks,\n            }),\n        }\n    }","sourceCodeStart":123,"sourceCodeEnd":159,"githubUrl":"https://github.com/quickwit-oss/quickwit/blob/a39730c5cdcd1a4fe798403737ae293999ea21f8/quickwit/quickwit-common/src/thread_pool/with_priority.rs#L123-L159","documentation":"This `.expect(\"failed to spawn thread pool\")` panic occurs in `ThreadPoolWithPriority::new` (quickwit-common/src/thread_pool/with_priority.rs) when rayon's `ThreadPoolBuilder::build()` returns `None`, i.e. rayon could not spawn the requested worker threads. As with the simple pool, this indicates an OS-level failure to create threads (resource limits, memory), and the constructor panics because a thread pool without workers cannot function.","triggerScenarios":"Creating a `ThreadPoolWithPriority` when the process has exhausted its allowed thread count (RLIMIT_NPROC, cgroup pids.max, container limits) or thread creation fails due to memory pressure; also possible if a num_threads value rayon cannot honor is passed.","commonSituations":"Deployments inside containers with restrictive pids cgroup controllers, hosts with very low `ulimit -u`, or environments where many pools/services each spawn threads until the budget is gone.","solutions":["Increase the process thread limit (`ulimit -u`, systemd TasksMax, Docker/K8s pids limit) and restart the service.","Lower the pool's `num_threads` configuration so rayon can spawn the requested workers.","Check memory availability; out-of-memory conditions can make thread spawning fail.","Reduce the number of concurrently created pools or total threads across the application."],"exampleFix":"// before: Kubernetes pod with tight pids limit\nresources:\n  limits:\n    pids: 50\n// after: give headroom for all rayon pools\nresources:\n  limits:\n    pids: 512","handlingStrategy":"validation","validationCode":"// Verify OS thread budget before building the pool\nlet (soft, _) = nix::sys::resource::getrlimit(nix::sys::resource::Resource::RLIMIT_NPROC)?;\nlet cur_threads: u64 = std::fs::read_to_string(\"/proc/self/status\")?\n    .lines().find_map(|l| l.strip_prefix(\"Threads:\")?.trim().parse().ok()).unwrap_or(0);\nassert!(cur_threads + num_threads as u64 <= soft, \"thread limit too low for pool\");","typeGuard":"fn thread_budget_available(extra: usize) -> bool {\n    std::fs::read_to_string(\"/proc/self/status\").ok().and_then(|s| {\n        let cur: u64 = s.lines().find_map(|l| l.strip_prefix(\"Threads:\")?.trim().parse().ok())?;\n        let (soft, _) = nix::sys::resource::getrlimit(nix::sys::resource::Resource::RLIMIT_NPROC).ok()?;\n        Some(cur + extra as u64 <= soft)\n    }).unwrap_or(true)\n}","tryCatchPattern":"let pool = std::panic::catch_unwind(|| ThreadPoolWithPriority::new(\"merge\", n, prio))\n    .map_err(|_| anyhow::anyhow!(\"thread pool init failed: raise pids limit or lower num_threads\"))?;","preventionTips":["Raise Kubernetes pids limits / systemd TasksMax for services spawning rayon pools.","Keep total threads across all pools below the OS limit.","Check memory pressure — thread creation fails under OOM.","Fail fast at startup with a clear config check rather than at pool construction."],"tags":["rust","rayon","thread-pool","resource-limit","startup"],"backgroundTag":"module-init-failed","analyzedSha":"a39730c5cdcd1a4fe798403737ae293999ea21f8","analyzedAt":"2026-09-08T13:19:37.784Z","contentChangedAt":"2026-09-08T13:19:37.784Z","schemaVersion":2},"datasetVersion":"2026-09-14T11:17:12.474Z"}