tracel-ai/burn · error

training worker on device {device_id} failed: {msg}

Error message

training worker on device {device_id} failed: {msg}

What it means

The distributed/multi-device training step aborts because a training worker on the given device returned an error message `msg` (e.g., the model forward/backward failed or the worker channel closed). The step cannot produce outputs for the batch, so the error from the worker is propagated to the caller.

Source

Thrown at crates/burn-train/src/learner/supervised/step/train.rs:182

                Some(Ok(item)) => {
                    worker.register(item, model);
                    num_send += 1;
                    let progress = dataloader.progress();
                    items_total += progress.items_total;
                    items_processed += progress.items_processed;
                }
                Some(Err(err)) => return Err(err),
                None => {}
            }
        }

        let mut outputs = Vec::with_capacity(num_send);

        for _ in 0..num_send {
            match self.receiver.recv().unwrap() {
                WorkerMessage::Output(output) => outputs.push(output),
                WorkerMessage::Error(device_id, msg) => {
                    panic!("training worker on device {device_id} failed: {msg}");
                }
            }
        }

        Ok((outputs, Progress::new(items_processed, items_total, unit)))
    }
}

View on GitHub (pinned to d16f7ba2ed)

Solutions

1. Read `msg` for the underlying worker failure (OOM, device fault, shape mismatch)
2. Reduce batch size if the worker ran out of device memory
3. Verify the device is healthy and accessible
4. Ensure the dataloader and model run correctly on that device in isolation

When it happens

Trigger: Thrown at crates/burn-train/src/learner/supervised/step/train.rs:182 when the library encounters an invalid state.

Common situations: See trigger scenarios.

AI-assisted analysis of tracel-ai/burn@d16f7ba2ed (2026-09-05). Data as JSON: /api/errors/2dd6ca2ae6e53557. Report an issue: GitHub.