What causes "Distributed data parallel worker {id} failed: {msg}" in tracel-ai/burn?

Inspect the original panic text after the colon and the worker id to identify which device failed

Distributed data parallel worker {id} failed: {msg}

Error message

Distributed data parallel worker {id} failed: {msg}

What it means

This panic fires in the DDP training strategy's `fit` when a secondary (non-main) worker thread, identified by its peer id, terminated with an `Err` JoinError — i.e. that worker panicked. Burn propagates the worker's original panic message; in DDP the failure of any peer invalidates the whole synchronized training run, so the learner aborts.

Source

Thrown at crates/burn-train/src/learner/supervised/strategies/ddp/strategy.rs:163

            thread::spawn(move || {
                tx.send((MAIN_ID, main_handle.join())).ok();
            });
        }
        drop(result_tx);

        let mut main_model = None;
        for _ in 0..peer_count {
            match result_rx
                .recv()
                .expect("worker reaper thread disconnected unexpectedly")
            {
                (MAIN_ID, Ok(model)) => main_model = Some(model),
                (id, Err(payload)) => {
                    let msg = panic_message(payload.as_ref());
                    if id == MAIN_ID {
                        panic!("Distributed data parallel main worker failed: {msg}");
                    } else {
                        panic!("Distributed data parallel worker {id} failed: {msg}");
                    }
                }
                (_, Ok(_)) => {}
            }
        }
        // Main worker had the event processor
        let model = main_model.expect("main worker should have produced a model");

        if interrupter.should_stop() {
            let reason = interrupter
                .get_message()
                .unwrap_or(String::from("Reason unknown"));
            log::info!("Training interrupted: {reason}");
        }
        let Ok(event_processor) = Arc::try_unwrap(event_processor) else {
            panic!("Event processor still held!");
        };
        let Ok(event_processor) = event_processor.into_inner() else {

View on GitHub (pinned to d16f7ba2ed)

Solutions

Inspect the original panic text after the colon and the worker id to identify which device failed
Check that every device in the strategy's devices list is distinct, exists, and has free memory
Verify each device's dataloader produces at least one batch (non-empty, divisible dataset/shards)
Re-run with a single device or fewer devices to isolate the failing device and reproduce the underlying panic
Update burn/backend crates; known per-device backend panics are frequently fixed upstream

Example fix

// before
DistributedDataParallelStrategy::new(&[device0, device5]) // device5 does not exist
// after
DistributedDataParallelStrategy::new(&[device0, device1]) // all listed devices valid

Defensive patterns

Strategy: try-catch

Validate before calling

fn validate_devices(devices: &[B]) -> Result<(), String> {
    if devices.is_empty() { return Err("no devices".into()); }
    let mut seen = std::collections::HashSet::new();
    for d in devices {
        if !seen.insert(format("{d:?}")) {
            return Err(format!("duplicate device {d:?}"));
        }
    }
    Ok(())
}

Try / catch

let result = std::panic::catch_unwind(std::panic::AssertUnwindSafe(||
    learner.fit(devices.clone(), dataloader_train, dataloader_valid)
));
match result {
    Ok(output) => output,
    Err(payload) => {
        let msg = payload.downcast_ref::<String>().cloned()
            .or_else(|| payload.downcast_ref::<&str>().map(|s| s.to_string()))
            .unwrap_or_default();
        eprintln!("DDP worker failed: {msg}"); // includes the failing worker id
        // degrade gracefully: retry on the main device only
        learner.fit(devices[0].clone(), dataloader_train, dataloader_valid)
    }
}

Prevention

Verify every device index in the devices list exists and is idle/available before launching DDP
Make sure the dataset splits evenly so every peer's dataloader is non-empty
Run a quick single-device pass first to catch panics unrelated to distribution
Avoid panicking custom metrics/handlers that run inside worker threads
Keep backend drivers (GPU drivers, wgpu adapters, CUDA/torch libs) up to date and tested

When it happens

Trigger: Running `Learner::fit` with `DistributedDataParallelStrategy` where one of the devices listed after the first panics during training — e.g. a dataloader error on that device's training shard, backend panic, or OOM on that device — detected when the supervisor receives `(id, Err(payload))` from the reaper thread.

Common situations: Multi-GPU jobs where a secondary GPU is misconfigured, busy, or out of memory; uneven dataloaders where one shard yields zero batches; a backend (wgpu/cubecl/tch) that fails on the specific device index; custom metric code that panics on data seen only by that peer.

Related errors

AI-assisted analysis of tracel-ai/burn@d16f7ba2ed (2026-09-05). Data as JSON: /api/errors/04f90f9d86247597. Report an issue: GitHub.