What causes "Failed to spawn session worker thread" in tracel-ai/burn?

Reduce concurrent sessions or raise the thread/pids limit (ulimit -u, cgroup pids.max)

Failed to spawn session worker thread

Error message

Failed to spawn session worker thread

What it means

The session worker spawns a dedicated OS thread per remote session that block_on's the session's async run loop. The expect panics if std::thread::Builder::spawn fails, i.e. the OS refused to create a new thread.

Source

Thrown at crates/burn-remote/src/server/worker.rs:126

            transfer,
            local_comm,
            graphs: Mutex::new(HashMap::new()),
            probe,
        };
        let (sender, receiver) = mpsc::channel(TASK_CHANNEL_CAPACITY);
        handler.drive(receiver);
        sender
    }

    // Must be called from within the runtime that owns the endpoint: it captures `Handle::current`.
    #[cfg(not(target_family = "wasm"))]
    fn drive(self, receiver: mpsc::Receiver<Task>) {
        let handle = Handle::current();
        let session_id = self.session_id;
        std::thread::Builder::new()
            .name(format!("burn-remote-session-{session_id}"))
            .spawn(move || handle.block_on(self.run(receiver)))
            .expect("Failed to spawn session worker thread");
    }

    #[cfg(target_family = "wasm")]
    fn drive(self, receiver: mpsc::Receiver<Task>) {
        spawn_detached(self.run(receiver));
    }

    /// Drain the task channel, running each task to completion in arrival order, then tear the
    /// session down in an order that releases its memory.
    async fn run(mut self, mut receiver: mpsc::Receiver<Task>) {
        let session_id = self.session_id;

        log::debug!("Session {session_id} worker started");
        while let Some(task) = receiver.recv().await {
            if let Err(err) = self.process_task(task).await {
                // One task failing doesn't tear down the session: read/sync/dtype failures surface
                // to the client through their response, fire-and-forget failures are logged here,
                // and the worker keeps processing subsequent tasks.

View on GitHub (pinned to d16f7ba2ed)

Solutions

Reduce concurrent sessions or raise the thread/pids limit (ulimit -u, cgroup pids.max)
Fix leaks: ensure sessions are closed so their worker threads exit
Increase container memory; thread stack allocation may fail under low memory
Patch/handle spawn failure gracefully instead of expect if graceful degradation is desired

Example fix

// before
.spawn(move || handle.block_on(self.run(receiver)))
.expect("Failed to spawn session worker thread");
// after
// system-level: raise limits, e.g. `ulimit -u 4096` or set pids.max in the cgroup

Defensive patterns

Strategy: retry

Validate before calling

// Sanity-check spawn capacity before opening sessions
std::thread::Builder::new().name("probe").spawn(|| {}).map_err(|e| anyhow!("cannot spawn threads: {e}"))?;

Try / catch

match std::thread::Builder::new().name(name).spawn(work) {
    Ok(h) => h,
    Err(e) => return Err(anyhow!("session worker spawn failed: {e}")),
}

Prevention

Cap concurrent remote sessions; each spawns a dedicated thread
Raise RLIMIT_NPROC / cgroup pids.max for the server process
Ensure sessions are properly shut down so worker threads don't leak

When it happens

Trigger: Creating a new remote session when the process/system cannot spawn a thread — thread-count limits (ulimit, cgroup pids.max), memory exhaustion, or too many concurrent sessions.

Common situations: Many simultaneous remote clients each opening a session on a constrained container; hitting RLIMIT_NPROC; leaks of previous session threads exhausting resources.

Related errors

AI-assisted analysis of tracel-ai/burn@d16f7ba2ed (2026-09-05). Data as JSON: /api/errors/f7a4dfcc854ef681. Report an issue: GitHub.