quickwit-oss/quickwit · critical

failed to spawn thread pool

Error message

failed to spawn thread pool

What it means

This `.expect("failed to spawn thread pool")` panic occurs in `SimpleThreadPool::new` (quickwit-common/src/thread_pool/simple.rs) when the underlying rayon `ThreadPoolBuilder::build()` returns `None`. Rayon fails to build a pool when it cannot spawn the requested worker threads (or the requested thread count is 0), typically due to operating-system resource limits. Since a thread pool with no workers is unusable, the constructor aborts by panicking.

Source

Thrown at quickwit/quickwit-common/src/thread_pool/simple.rs:47

struct SimpleThreadPool {
    thread_pool: Arc<rayon::ThreadPool>,
    ongoing_tasks: Gauge,
    pending_tasks: Gauge,
}

impl SimpleThreadPool {
    fn new(name: &'static str, num_threads_opt: Option<usize>) -> SimpleThreadPool {
        let mut rayon_pool_builder = rayon::ThreadPoolBuilder::new()
            .thread_name(move |thread_id| format!("quickwit-{name}-{thread_id}"))
            .panic_handler(move |_my_panic| {
                error!("task running in the quickwit {name} thread pool panicked");
            });
        if let Some(num_threads) = num_threads_opt {
            rayon_pool_builder = rayon_pool_builder.num_threads(num_threads);
        }
        let thread_pool = rayon_pool_builder
            .build()
            .expect("failed to spawn thread pool");
        let labels = labels!("pool" => name);
        let ongoing_tasks = gauge!(parent: THREAD_POOL_ONGOING_TASKS, labels: [labels]);
        let pending_tasks = gauge!(parent: THREAD_POOL_PENDING_TASKS, labels: [labels]);
        SimpleThreadPool {
            thread_pool: Arc::new(thread_pool),
            ongoing_tasks,
            pending_tasks,
        }
    }

    /// Function similar to `tokio::spawn_blocking`.
    ///
    /// Here are two important differences however:
    ///
    /// 1) The task runs on a rayon thread pool managed by Quickwit. This pool is specifically used
    ///    only to run CPU-intensive work and is configured to contain `num_cpus` cores.
    ///
    /// 2) Before the task is effectively scheduled, we check that the spawner is still interested

View on GitHub (pinned to a39730c5cd)

Solutions

1. Raise the OS thread limits: increase `ulimit -u` (RLIMIT_NPROC) or the container/Kubernetes `pids.max` limit and restart.
2. Reduce the configured `num_threads` for the pool (or unset it to use num_cpus) so thread spawning succeeds within current limits.
3. Check host memory/pressure: thread creation can fail under memory exhaustion; free memory or add headroom.
4. If running many pools, consolidate pools or lower total worker counts so the cumulative thread count fits the limit.

Example fix

// before (config asking for too many threads under a pids limit)
[searcher]
concurrency = 1024
// after: lower concurrency / let it default
[searcher]
concurrency = 8
# plus, at OS level:
# ulimit -u 4096  (or raise the pod pids limit)

// Check thread headroom before constructing the pool (Linux)
use std::fs;
let tasks = fs::read_to_string("/proc/self/status")
    .ok()
    .and_then(|s| s.lines().find(|l| l.starts_with("Threads:"))?.split_whitespace().last()?.parse::<u64>().ok())
    .unwrap_or(0);
let limit = nix::sys::resource::getrlimit(nix::sys::resource::Resource::RLIMIT_NPROC)
    .ok().map(|(soft, _)| soft).unwrap_or(u64::MAX);
assert!(tasks + requested_threads < limit, "insufficient thread headroom");

fn can_spawn_threads(requested: usize) -> bool {
    // heuristic: current process threads + requested below rlimit
    fs::read_to_string("/proc/self/status").map(|s| {
        let cur: u64 = s.lines().find_map(|l| l.strip_prefix("Threads:")?.trim().parse().ok()).unwrap_or(0);
        let (soft, _) = nix::sys::resource::getrlimit(nix::sys::resource::Resource::RLIMIT_NPROC).ok()?;
        cur + requested as u64 < soft
    }).unwrap_or(true)
}

// Constructor panics; wrap pool creation at startup
let pool = std::panic::catch_unwind(|| SimpleThreadPool::new("search", n, throttle))
    .map_err(|_| anyhow::anyhow!("failed to spawn thread pool; check ulimit/pids limit"))?;

When it happens

Trigger: Constructing a `SimpleThreadPool` via `new(name, num_threads, ...)` when rayon cannot spawn the requested threads: process/thread limit (`RLIMIT_NPROC`, `ulimit -u`, cgroup pids.max, container thread limits) reached, memory exhaustion at thread creation, or configuring `num_threads` such that rayon refuses it.

Common situations: Running Quickwit in tightly constrained Docker/Kubernetes pods with low `pids` limits or low `RLIMIT_NPROC`; heavily loaded hosts that already exhaust the thread budget; misconfigured pool sizing in `quickwit.yaml` requesting an impossible thread count.

Related errors

• failed to spawn thread pool
• compactor service enabled but no compaction client available
• node not found in pending
• OTP logs or traces do not support VRL transforms
• position of a Kafka partition should never be EOF

AI-assisted analysis of quickwit-oss/quickwit@a39730c5cd (2026-09-08). Data as JSON: /api/errors/e09f3018ee8e8273. Report an issue: GitHub.