tracel-ai/burn · error

Dataloader worker thread should be alive

Error message

Dataloader worker thread should be alive

What it means

During Dataloader iteration, the main thread sends commands to worker threads over channels; the expect fires when a worker's channel sender is disconnected, meaning that worker thread has already terminated. The library assumes all workers stay alive for the iterator's lifetime, so a dead worker is an invariant violation.

Source

Thrown at crates/burn-core/src/data/dataloader/multithread.rs:255

}

impl<I, O> DataLoader<O> for MultiThreadDataLoader<I, O>
where
    I: Send + Sync + Clone + 'static,
    O: Send + 'static + std::fmt::Debug,
{
    fn iter<'a>(&'a self) -> Box<dyn DataLoaderIterator<O> + 'a> {
        let workers = self.workers();

        let (sender, receiver) = mpsc::sync_channel::<Message<O>>(MAX_QUEUED_ITEMS);
        let unit: Option<String> = Some("items".to_string());

        let mut progresses = Vec::with_capacity(workers.senders.len());
        for (command_sender, &num_items) in workers.senders.iter().zip(workers.item_counts.iter()) {
            progresses.push(Progress::new(0, num_items, unit.clone()));
            command_sender
                .send(sender.clone())
                .expect("Dataloader worker thread should be alive");
        }
        let num_workers = workers.senders.len();

        // Drop our sender so the channel disconnects once every worker is done.
        drop(sender);

        Box::new(MultiThreadsDataloaderIterator::new(
            receiver,
            num_workers,
            progresses,
        ))
    }

    fn num_items(&self) -> usize {
        // For num_items, we can directly use the dataset size without
        // necessarily initializing the full loader
        self.dataset.len()
    }

View on GitHub (pinned to d16f7ba2ed)

Solutions

1. Fix the panic inside the worker (usually in the dataset indexing or collation closure) — check logs for the original worker panic message.
2. Recreate the Dataloader instead of reusing it after a worker failure.
3. Reduce memory pressure (smaller batch size, fewer workers) if workers were killed by OOM.

Example fix

// before: dataset closure that can panic inside worker
let ds = Mapper::new(base, |item| item.parse().unwrap());
// after
let ds = Mapper::new(base, |item| item.parse().unwrap_or_default());

// Wrap dataloader iteration and recreate on worker death
match std::panic::catch_unwind(AssertUnwindSafe(|| {
    for batch in dataloader.iter() { /* consume */ }
})) {
    Ok(_) => {},
    Err(_) => dataloader = build_dataloader(); // workers died; rebuild
}

When it happens

Trigger: Calling iter() on a multi-threaded Dataloader after a worker thread panicked (e.g. its map/collate closure panicked) or was killed, so command_sender.send() fails because the receiver was dropped.

Common situations: User-supplied transforms/bug in dataset items that panic inside a worker; OOM killer terminating a worker thread; stopping the dataloader and re-iterating a workers handle in an inconsistent state.

Related errors

• capture tensor operations must run inside CaptureDevice::cap
• Capture tensors do not support autodiff
• Autodiff should not wrap an autodiff tensor.
• Requires autodiff tensor.
• Distributed operations are not supported for tensor kind {ot

AI-assisted analysis of tracel-ai/burn@d16f7ba2ed (2026-09-05). Data as JSON: /api/errors/acbc6d0c81912438. Report an issue: GitHub.