{"record":{"id":"aa8db750184b8c0e","repo":"tracel-ai/burn","slug":"distributed-data-parallel-main-worker-failed-msg","errorCode":null,"errorMessage":"Distributed data parallel main worker failed: {msg}","messagePattern":"Distributed data parallel main worker failed: (.+?)","errorType":"panic","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"crates/burn-train/src/learner/supervised/strategies/ddp/strategy.rs","lineNumber":161,"sourceCode":"        {\n            let tx = result_tx.clone();\n            thread::spawn(move || {\n                tx.send((MAIN_ID, main_handle.join())).ok();\n            });\n        }\n        drop(result_tx);\n\n        let mut main_model = None;\n        for _ in 0..peer_count {\n            match result_rx\n                .recv()\n                .expect(\"worker reaper thread disconnected unexpectedly\")\n            {\n                (MAIN_ID, Ok(model)) => main_model = Some(model),\n                (id, Err(payload)) => {\n                    let msg = panic_message(payload.as_ref());\n                    if id == MAIN_ID {\n                        panic!(\"Distributed data parallel main worker failed: {msg}\");\n                    } else {\n                        panic!(\"Distributed data parallel worker {id} failed: {msg}\");\n                    }\n                }\n                (_, Ok(_)) => {}\n            }\n        }\n        // Main worker had the event processor\n        let model = main_model.expect(\"main worker should have produced a model\");\n\n        if interrupter.should_stop() {\n            let reason = interrupter\n                .get_message()\n                .unwrap_or(String::from(\"Reason unknown\"));\n            log::info!(\"Training interrupted: {reason}\");\n        }\n        let Ok(event_processor) = Arc::try_unwrap(event_processor) else {\n            panic!(\"Event processor still held!\");","sourceCodeStart":143,"sourceCodeEnd":179,"githubUrl":"https://github.com/tracel-ai/burn/blob/d16f7ba2ed0d41408189384044cc886fb4c8f957/crates/burn-train/src/learner/supervised/strategies/ddp/strategy.rs#L143-L179","documentation":"This panic fires in the DDP (multi-device) training strategy's `fit` when the MAIN worker thread (peer id `MAIN_ID`) finished with an `Err` JoinError, i.e. the main worker thread panicked. Burn re-raises the worker's original panic message wrapped in this message, because the supervisor loop in the learner cannot resume training if the main device (which owns the event processor and returns the final model) died.","triggerScenarios":"Running `Learner::fit` with `DistributedDataParallelStrategy` where the main worker thread panics mid-training — e.g. a backend/CUDA/wGPU panic on the main device, a dataloader error on the first training dataloader, a panic in a metric/evaluator running on the main device, or OOM on the main GPU.","commonSituations":"Multi-GPU training where the primary GPU runs out of memory or a kernel panics; a bug or incompatible backend (tch/wgpu/cubecl) on the main device; misconfigured devices list (same device used twice); panics inside custom metrics, checkpoints or event handlers executing on the main worker.","solutions":["Read the original panic message embedded after the colon — it is the real error from the main worker thread and points to the failing component","Verify the main device (first entry of the devices list) is valid and has enough free memory; test with a smaller batch size","Run single-device training (`SingleDevice` strategy) on the same model/data to reproduce and debug the underlying panic without DDP","Check that the dataloader assigned to the main device works (dataset files present, num_workers settings valid)","Update burn and the backend crates to the latest compatible versions; backend panics are often already fixed upstream"],"exampleFix":"// before\nlearner.fit(device, dataloader, dataloader_valid); // devices: [gpu0, gpu0] -> main worker panics\n// after\nlearner.fit(device, dataloader, dataloader_valid); // devices: [gpu0, gpu1] distinct valid devices","handlingStrategy":"try-catch","validationCode":"fn validate_devices(devices: &[B]) -> Result<(), String> {\n    let mut seen = std::collections::HashSet::new();\n    for d in devices {\n        if !seen.insert(format(\"{d:?}\")) {\n            return Err(\"duplicate device in DDP devices list\".into());\n        }\n    }\n    Ok(())\n}","typeGuard":null,"tryCatchPattern":"let result = std::panic::catch_unwind(std::panic::AssertUnwindSafe(||\n    learner.fit(devices.clone(), dataloader_train, dataloader_valid)\n));\nmatch result {\n    Ok(output) => output,\n    Err(payload) => {\n        let msg = payload.downcast_ref::<String>().cloned()\n            .or_else(|| payload.downcast_ref::<&str>().map(|s| s.to_string()))\n            .unwrap_or_default();\n        eprintln!(\"DDP main worker failed: {msg}\"); // original cause is after the colon\n        // fall back to single-device training\n        learner.fit(devices[0].clone(), dataloader_train, dataloader_valid)\n    }\n}","preventionTips":["Ensure the devices list contains distinct, existing devices with sufficient free memory","Smoke-test training on a single device before scaling to DDP","Validate dataloaders yield at least one batch per device (non-empty, shardable dataset)","Keep custom metrics/checkpointers panic-free; avoid unwrap on device-dependent code paths","Pin and test known-good versions of burn and the backend (cubecl/wgpu/tch) crates"],"tags":["distributed-training","panic","multi-device","ddp"],"backgroundTag":"worker-thread-panicked","analyzedSha":"d16f7ba2ed0d41408189384044cc886fb4c8f957","analyzedAt":"2026-09-05T13:19:14.260Z","contentChangedAt":"2026-09-05T13:19:14.260Z","schemaVersion":2},"datasetVersion":"2026-09-12T17:17:11.597Z"}