{"record":{"id":"e629481935769f05","repo":"EpicGames/lore","slug":"could-not-spawn-payload-generator-thread","errorCode":null,"errorMessage":"could not spawn payload generator thread","messagePattern":"could not spawn payload generator thread","errorType":"panic","errorClass":null,"httpStatus":null,"severity":"error","filePath":"lore-revision/src/store/seeder.rs","lineNumber":156,"sourceCode":"            .spawn(move || {\n                let mut rng = rand::rng();\n                let mut buffer = BytesMut::with_capacity(FRAGMENT_SIZE_THRESHOLD);\n\n                for _ in 0..per_task_count {\n                    buffer.resize(FRAGMENT_SIZE_THRESHOLD, 0);\n                    rng.fill_bytes(&mut buffer);\n\n                    if let Err(e) = tx.blocking_send(buffer.split().freeze()) {\n                        // The receiver must've gone away (this shouldn't happen in a real world\n                        // scenario).\n                        lore_warn!(\"Failed to send fragment data: {e:?}\");\n                        return;\n                    }\n\n                    in_flight.fetch_add(1, Ordering::Relaxed);\n                }\n            })\n            .expect(\"could not spawn payload generator thread\");\n    }\n\n    // The generators hold the only remaining senders, so the receive loop below ends when the last\n    // of them finishes.\n    drop(tx);\n\n    lore_info!(\"Listening for payloads\");\n\n    let mut join_set = JoinSet::new();\n\n    // In local testing on a 16 core system we never exceeded 23 concurrent tasks, so this seems\n    // like a reasonable limit.\n    let semaphore = Arc::new(Semaphore::new(task_count * 2));\n\n    let written = Arc::new(AtomicUsize::new(0));\n    let processed = Arc::new(AtomicUsize::new(0));\n\n    while let Some(bytes) = rx.recv().await {","sourceCodeStart":138,"sourceCodeEnd":174,"githubUrl":"https://github.com/EpicGames/lore/blob/074eb0b0d1194c997d7cf28b55519e3e197b3e23/lore-revision/src/store/seeder.rs#L138-L174","documentation":"This panic comes from std::thread::Builder::scope-style spawn in do_seed_local_store: after registering the in-flight counter, the code calls .expect on the JoinHandle result of spawning a payload generator thread. The library panics rather than returning an error because local-store seeding cannot proceed without its worker threads. Thread spawn only fails when the OS refuses to create a new thread.","triggerScenarios":"Calling seed_local_store (which calls do_seed_local_store) while the process is out of resources for new threads: RLIMIT_NPROC/ulimit -u hit, pthread_create returning EAGAIN due to memory exhaustion (thread stack allocation failure), or spawning from an environment that forbids thread creation.","commonSituations":"Containers with low pid/thread limits (e.g. Docker pids-limit, Kubernetes pod limits), heavily loaded machines at the thread-count ceiling, memory-capped environments where the default 8MB stack cannot be allocated, or tests running hundreds of concurrent seed_local_store calls.","solutions":["Raise the process thread limit (ulimit -u, container pids-limit, cgroup pids.max) before running the server","Reduce concurrent seed_local_store calls so fewer generator threads exist at once","Check free memory; reduce thread stack usage or add swap so pthread_create can allocate the stack","If it persists, restructure the call site to run seeding on an existing runtime worker instead of relying on the library's spawn"],"exampleFix":"// before\n// ulimit -u 64  ./lore-server   (thread limit too low -> panic)\n// after\n// ulimit -u 4096 && ./lore-server","handlingStrategy":"retry","validationCode":"// check spawn headroom before calling seed_local_store\nlet max_threads = std::fs::read_to_string(\"/proc/sys/kernel/threads-max\").ok().and_then(|s| s.trim().parse::<i64>().ok());\nlet cur = std::fs::read_to_string(\"/proc/self/status\").ok().and_then(|s| s.lines().find(|l| l.starts_with(\"Threads:\")).and_then(|l| l.split_whitespace().nth(1).and_then(|n| n.parse::<i64>().ok())));\nassert!(max_threads.zip(cur).map_or(true, |(m, c)| m - c > 16), \"not enough thread headroom to seed\");","typeGuard":"fn can_spawn_more_threads(headroom_needed: i64) -> bool {\n    std::fs::read_to_string(\"/proc/loadavg\").is_ok() // on linux, additionally compare /proc/self/status Threads to ulimit\n}","tryCatchPattern":"// Rust panics cannot be caught by Result; isolate seeding in a spawned task and inspect JoinError\nmatch tokio::task::spawn_blocking(move || seed_local_store(...)).await {\n    Ok(Ok(())) => {},\n    Ok(Err(e)) => eprintln!(\"seed failed: {e}\"),\n    Err(join_err) => eprintln!(\"seeder panicked (likely thread spawn failure): {join_err}\"),\n}","preventionTips":["Raise container/pod pids limits before running services that spawn worker threads","Monitor thread counts in production and alert before hitting limits","Avoid calling seed_local_store concurrently from many tasks","Run load tests on memory-constrained environments to surface pthread_create EAGAIN early"],"tags":["rust","threading","spawn-failure","resource-limits"],"backgroundTag":"unsupported-platform","analyzedSha":"074eb0b0d1194c997d7cf28b55519e3e197b3e23","analyzedAt":"2026-09-13T09:00:57.509Z","contentChangedAt":"2026-09-13T09:00:57.509Z","schemaVersion":2},"datasetVersion":"2026-09-16T09:17:16.951Z"}