What causes "could not spawn payload generator thread" in EpicGames/lore?

Raise the process thread limit (ulimit -u, container pids-limit, cgroup pids.max) before running the server

could not spawn payload generator thread

Error message

could not spawn payload generator thread

What it means

This panic comes from std::thread::Builder::scope-style spawn in do_seed_local_store: after registering the in-flight counter, the code calls .expect on the JoinHandle result of spawning a payload generator thread. The library panics rather than returning an error because local-store seeding cannot proceed without its worker threads. Thread spawn only fails when the OS refuses to create a new thread.

Solutions

Raise the process thread limit (ulimit -u, container pids-limit, cgroup pids.max) before running the server
Reduce concurrent seed_local_store calls so fewer generator threads exist at once
Check free memory; reduce thread stack usage or add swap so pthread_create can allocate the stack
If it persists, restructure the call site to run seeding on an existing runtime worker instead of relying on the library's spawn

Example fix

// before
// ulimit -u 64  ./lore-server   (thread limit too low -> panic)
// after
// ulimit -u 4096 && ./lore-server

Defensive patterns

Strategy: retry

Validate before calling

// check spawn headroom before calling seed_local_store
let max_threads = std::fs::read_to_string("/proc/sys/kernel/threads-max").ok().and_then(|s| s.trim().parse::<i64>().ok());
let cur = std::fs::read_to_string("/proc/self/status").ok().and_then(|s| s.lines().find(|l| l.starts_with("Threads:")).and_then(|l| l.split_whitespace().nth(1).and_then(|n| n.parse::<i64>().ok())));
assert!(max_threads.zip(cur).map_or(true, |(m, c)| m - c > 16), "not enough thread headroom to seed");

Type guard

fn can_spawn_more_threads(headroom_needed: i64) -> bool {
    std::fs::read_to_string("/proc/loadavg").is_ok() // on linux, additionally compare /proc/self/status Threads to ulimit
}

Try / catch

// Rust panics cannot be caught by Result; isolate seeding in a spawned task and inspect JoinError
match tokio::task::spawn_blocking(move || seed_local_store(...)).await {
    Ok(Ok(())) => {},
    Ok(Err(e)) => eprintln!("seed failed: {e}"),
    Err(join_err) => eprintln!("seeder panicked (likely thread spawn failure): {join_err}"),
}

Prevention

Raise container/pod pids limits before running services that spawn worker threads
Monitor thread counts in production and alert before hitting limits
Avoid calling seed_local_store concurrently from many tasks
Run load tests on memory-constrained environments to surface pthread_create EAGAIN early

When it happens

Trigger: Calling seed_local_store (which calls do_seed_local_store) while the process is out of resources for new threads: RLIMIT_NPROC/ulimit -u hit, pthread_create returning EAGAIN due to memory exhaustion (thread stack allocation failure), or spawning from an environment that forbids thread creation.

Common situations: Containers with low pid/thread limits (e.g. Docker pids-limit, Kubernetes pod limits), heavily loaded machines at the thread-count ceiling, memory-capped environments where the default 8MB stack cannot be allocated, or tests running hundreds of concurrent seed_local_store calls.

Understand the failure class

Background: "unsupported platform" / "not supported on this platform" errors: what they mean and how to fix them — this error's family across 47 libraries.

Related errors

AI-assisted analysis of EpicGames/lore@074eb0b0d1 (2026-09-13). Data as JSON: /api/errors/e629481935769f05. Report an issue: GitHub.

Appendix: source

Thrown at lore-revision/src/store/seeder.rs:156

            .spawn(move || {
                let mut rng = rand::rng();
                let mut buffer = BytesMut::with_capacity(FRAGMENT_SIZE_THRESHOLD);

                for _ in 0..per_task_count {
                    buffer.resize(FRAGMENT_SIZE_THRESHOLD, 0);
                    rng.fill_bytes(&mut buffer);

                    if let Err(e) = tx.blocking_send(buffer.split().freeze()) {
                        // The receiver must've gone away (this shouldn't happen in a real world
                        // scenario).
                        lore_warn!("Failed to send fragment data: {e:?}");
                        return;
                    }

                    in_flight.fetch_add(1, Ordering::Relaxed);
                }
            })
            .expect("could not spawn payload generator thread");
    }

    // The generators hold the only remaining senders, so the receive loop below ends when the last
    // of them finishes.
    drop(tx);

    lore_info!("Listening for payloads");

    let mut join_set = JoinSet::new();

    // In local testing on a 16 core system we never exceeded 23 concurrent tasks, so this seems
    // like a reasonable limit.
    let semaphore = Arc::new(Semaphore::new(task_count * 2));

    let written = Arc::new(AtomicUsize::new(0));
    let processed = Arc::new(AtomicUsize::new(0));

    while let Some(bytes) = rx.recv().await {

View on GitHub (pinned to 074eb0b0d1)