tracel-ai/burn · critical

Distributed data parallel main worker failed: {msg}

Error message

Distributed data parallel main worker failed: {msg}

What it means

This panic fires in the DDP (multi-device) training strategy's `fit` when the MAIN worker thread (peer id `MAIN_ID`) finished with an `Err` JoinError, i.e. the main worker thread panicked. Burn re-raises the worker's original panic message wrapped in this message, because the supervisor loop in the learner cannot resume training if the main device (which owns the event processor and returns the final model) died.

Source

Thrown at crates/burn-train/src/learner/supervised/strategies/ddp/strategy.rs:161

        {
            let tx = result_tx.clone();
            thread::spawn(move || {
                tx.send((MAIN_ID, main_handle.join())).ok();
            });
        }
        drop(result_tx);

        let mut main_model = None;
        for _ in 0..peer_count {
            match result_rx
                .recv()
                .expect("worker reaper thread disconnected unexpectedly")
            {
                (MAIN_ID, Ok(model)) => main_model = Some(model),
                (id, Err(payload)) => {
                    let msg = panic_message(payload.as_ref());
                    if id == MAIN_ID {
                        panic!("Distributed data parallel main worker failed: {msg}");
                    } else {
                        panic!("Distributed data parallel worker {id} failed: {msg}");
                    }
                }
                (_, Ok(_)) => {}
            }
        }
        // Main worker had the event processor
        let model = main_model.expect("main worker should have produced a model");

        if interrupter.should_stop() {
            let reason = interrupter
                .get_message()
                .unwrap_or(String::from("Reason unknown"));
            log::info!("Training interrupted: {reason}");
        }
        let Ok(event_processor) = Arc::try_unwrap(event_processor) else {
            panic!("Event processor still held!");

View on GitHub (pinned to d16f7ba2ed)

Solutions

1. Read the original panic message embedded after the colon — it is the real error from the main worker thread and points to the failing component
2. Verify the main device (first entry of the devices list) is valid and has enough free memory; test with a smaller batch size
3. Run single-device training (`SingleDevice` strategy) on the same model/data to reproduce and debug the underlying panic without DDP
4. Check that the dataloader assigned to the main device works (dataset files present, num_workers settings valid)
5. Update burn and the backend crates to the latest compatible versions; backend panics are often already fixed upstream

Example fix

// before
learner.fit(device, dataloader, dataloader_valid); // devices: [gpu0, gpu0] -> main worker panics
// after
learner.fit(device, dataloader, dataloader_valid); // devices: [gpu0, gpu1] distinct valid devices

fn validate_devices(devices: &[B]) -> Result<(), String> {
    let mut seen = std::collections::HashSet::new();
    for d in devices {
        if !seen.insert(format("{d:?}")) {
            return Err("duplicate device in DDP devices list".into());
        }
    }
    Ok(())
}

let result = std::panic::catch_unwind(std::panic::AssertUnwindSafe(||
    learner.fit(devices.clone(), dataloader_train, dataloader_valid)
));
match result {
    Ok(output) => output,
    Err(payload) => {
        let msg = payload.downcast_ref::<String>().cloned()
            .or_else(|| payload.downcast_ref::<&str>().map(|s| s.to_string()))
            .unwrap_or_default();
        eprintln!("DDP main worker failed: {msg}"); // original cause is after the colon
        // fall back to single-device training
        learner.fit(devices[0].clone(), dataloader_train, dataloader_valid)
    }
}

When it happens

Trigger: Running `Learner::fit` with `DistributedDataParallelStrategy` where the main worker thread panics mid-training — e.g. a backend/CUDA/wGPU panic on the main device, a dataloader error on the first training dataloader, a panic in a metric/evaluator running on the main device, or OOM on the main GPU.

Common situations: Multi-GPU training where the primary GPU runs out of memory or a kernel panics; a bug or incompatible backend (tch/wgpu/cubecl) on the main device; misconfigured devices list (same device used twice); panics inside custom metrics, checkpoints or event handlers executing on the main worker.

Related errors

• Distributed data parallel worker {id} failed: {msg}
• SVD fallback failed: {err}
• Quantization scheme is not valid for dtype {other:?}
• Can't store native sub-byte values
• {other:?} doesn't support native packing

AI-assisted analysis of tracel-ai/burn@d16f7ba2ed (2026-09-05). Data as JSON: /api/errors/aa8db750184b8c0e. Report an issue: GitHub.