quickwit-oss/quickwit · critical

failed to spawn thread pool

Error message

failed to spawn thread pool

What it means

This `.expect("failed to spawn thread pool")` panic occurs in `ThreadPoolWithPriority::new` (quickwit-common/src/thread_pool/with_priority.rs) when rayon's `ThreadPoolBuilder::build()` returns `None`, i.e. rayon could not spawn the requested worker threads. As with the simple pool, this indicates an OS-level failure to create threads (resource limits, memory), and the constructor panics because a thread pool without workers cannot function.

Source

Thrown at quickwit/quickwit-common/src/thread_pool/with_priority.rs:141

    /// The default priority.
    Normal,
    /// A high-priority task is scheduled before normal-priority tasks that are still pending.
    High,
}

impl ThreadPoolWithPriority {
    pub fn new(name: &'static str, num_threads_opt: Option<usize>) -> ThreadPoolWithPriority {
        let mut rayon_pool_builder = rayon::ThreadPoolBuilder::new()
            .thread_name(move |thread_id| format!("quickwit-{name}-{thread_id}"))
            .panic_handler(move |_my_panic| {
                error!("task running in the quickwit {name} thread pool panicked");
            });
        if let Some(num_threads) = num_threads_opt {
            rayon_pool_builder = rayon_pool_builder.num_threads(num_threads);
        }
        let thread_pool = rayon_pool_builder
            .build()
            .expect("failed to spawn thread pool");
        let max_running_tasks = thread_pool.current_num_threads();
        let labels = labels!("pool" => name);
        let ongoing_tasks = gauge!(parent: THREAD_POOL_ONGOING_TASKS, labels: [labels]);
        let pending_tasks = gauge!(parent: THREAD_POOL_PENDING_TASKS, labels: [labels]);
        ThreadPoolWithPriority {
            inner: Arc::new(ThreadPoolInner {
                thread_pool: Arc::new(thread_pool),
                max_running_tasks,
                num_running_tasks: AtomicUsize::new(0),
                state: Mutex::new(State {
                    high_priority_tasks: VecDeque::new(),
                    normal_priority_tasks: VecDeque::new(),
                }),
                ongoing_tasks,
                pending_tasks,
            }),
        }
    }

View on GitHub (pinned to a39730c5cd)

Solutions

1. Increase the process thread limit (`ulimit -u`, systemd TasksMax, Docker/K8s pids limit) and restart the service.
2. Lower the pool's `num_threads` configuration so rayon can spawn the requested workers.
3. Check memory availability; out-of-memory conditions can make thread spawning fail.
4. Reduce the number of concurrently created pools or total threads across the application.

Example fix

// before: Kubernetes pod with tight pids limit
resources:
  limits:
    pids: 50
// after: give headroom for all rayon pools
resources:
  limits:
    pids: 512

// Verify OS thread budget before building the pool
let (soft, _) = nix::sys::resource::getrlimit(nix::sys::resource::Resource::RLIMIT_NPROC)?;
let cur_threads: u64 = std::fs::read_to_string("/proc/self/status")?
    .lines().find_map(|l| l.strip_prefix("Threads:")?.trim().parse().ok()).unwrap_or(0);
assert!(cur_threads + num_threads as u64 <= soft, "thread limit too low for pool");

fn thread_budget_available(extra: usize) -> bool {
    std::fs::read_to_string("/proc/self/status").ok().and_then(|s| {
        let cur: u64 = s.lines().find_map(|l| l.strip_prefix("Threads:")?.trim().parse().ok())?;
        let (soft, _) = nix::sys::resource::getrlimit(nix::sys::resource::Resource::RLIMIT_NPROC).ok()?;
        Some(cur + extra as u64 <= soft)
    }).unwrap_or(true)
}

let pool = std::panic::catch_unwind(|| ThreadPoolWithPriority::new("merge", n, prio))
    .map_err(|_| anyhow::anyhow!("thread pool init failed: raise pids limit or lower num_threads"))?;

When it happens

Trigger: Creating a `ThreadPoolWithPriority` when the process has exhausted its allowed thread count (RLIMIT_NPROC, cgroup pids.max, container limits) or thread creation fails due to memory pressure; also possible if a num_threads value rayon cannot honor is passed.

Common situations: Deployments inside containers with restrictive pids cgroup controllers, hosts with very low `ulimit -u`, or environments where many pools/services each spawn threads until the budget is gone.

Related errors

• failed to spawn thread pool
• compactor service enabled but no compaction client available
• node not found in pending
• OTP logs or traces do not support VRL transforms
• position of a Kafka partition should never be EOF

AI-assisted analysis of quickwit-oss/quickwit@a39730c5cd (2026-09-08). Data as JSON: /api/errors/77f281bf7ae91710. Report an issue: GitHub.