{"record":{"id":"04f90f9d86247597","repo":"tracel-ai/burn","slug":"distributed-data-parallel-worker-id-failed-msg","errorCode":null,"errorMessage":"Distributed data parallel worker {id} failed: {msg}","messagePattern":"Distributed data parallel worker (.+?) failed: (.+?)","errorType":"panic","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"crates/burn-train/src/learner/supervised/strategies/ddp/strategy.rs","lineNumber":163,"sourceCode":"            thread::spawn(move || {\n                tx.send((MAIN_ID, main_handle.join())).ok();\n            });\n        }\n        drop(result_tx);\n\n        let mut main_model = None;\n        for _ in 0..peer_count {\n            match result_rx\n                .recv()\n                .expect(\"worker reaper thread disconnected unexpectedly\")\n            {\n                (MAIN_ID, Ok(model)) => main_model = Some(model),\n                (id, Err(payload)) => {\n                    let msg = panic_message(payload.as_ref());\n                    if id == MAIN_ID {\n                        panic!(\"Distributed data parallel main worker failed: {msg}\");\n                    } else {\n                        panic!(\"Distributed data parallel worker {id} failed: {msg}\");\n                    }\n                }\n                (_, Ok(_)) => {}\n            }\n        }\n        // Main worker had the event processor\n        let model = main_model.expect(\"main worker should have produced a model\");\n\n        if interrupter.should_stop() {\n            let reason = interrupter\n                .get_message()\n                .unwrap_or(String::from(\"Reason unknown\"));\n            log::info!(\"Training interrupted: {reason}\");\n        }\n        let Ok(event_processor) = Arc::try_unwrap(event_processor) else {\n            panic!(\"Event processor still held!\");\n        };\n        let Ok(event_processor) = event_processor.into_inner() else {","sourceCodeStart":145,"sourceCodeEnd":181,"githubUrl":"https://github.com/tracel-ai/burn/blob/d16f7ba2ed0d41408189384044cc886fb4c8f957/crates/burn-train/src/learner/supervised/strategies/ddp/strategy.rs#L145-L181","documentation":"This panic fires in the DDP training strategy's `fit` when a secondary (non-main) worker thread, identified by its peer id, terminated with an `Err` JoinError — i.e. that worker panicked. Burn propagates the worker's original panic message; in DDP the failure of any peer invalidates the whole synchronized training run, so the learner aborts.","triggerScenarios":"Running `Learner::fit` with `DistributedDataParallelStrategy` where one of the devices listed after the first panics during training — e.g. a dataloader error on that device's training shard, backend panic, or OOM on that device — detected when the supervisor receives `(id, Err(payload))` from the reaper thread.","commonSituations":"Multi-GPU jobs where a secondary GPU is misconfigured, busy, or out of memory; uneven dataloaders where one shard yields zero batches; a backend (wgpu/cubecl/tch) that fails on the specific device index; custom metric code that panics on data seen only by that peer.","solutions":["Inspect the original panic text after the colon and the worker id to identify which device failed","Check that every device in the strategy's devices list is distinct, exists, and has free memory","Verify each device's dataloader produces at least one batch (non-empty, divisible dataset/shards)","Re-run with a single device or fewer devices to isolate the failing device and reproduce the underlying panic","Update burn/backend crates; known per-device backend panics are frequently fixed upstream"],"exampleFix":"// before\nDistributedDataParallelStrategy::new(&[device0, device5]) // device5 does not exist\n// after\nDistributedDataParallelStrategy::new(&[device0, device1]) // all listed devices valid","handlingStrategy":"try-catch","validationCode":"fn validate_devices(devices: &[B]) -> Result<(), String> {\n    if devices.is_empty() { return Err(\"no devices\".into()); }\n    let mut seen = std::collections::HashSet::new();\n    for d in devices {\n        if !seen.insert(format(\"{d:?}\")) {\n            return Err(format!(\"duplicate device {d:?}\"));\n        }\n    }\n    Ok(())\n}","typeGuard":null,"tryCatchPattern":"let result = std::panic::catch_unwind(std::panic::AssertUnwindSafe(||\n    learner.fit(devices.clone(), dataloader_train, dataloader_valid)\n));\nmatch result {\n    Ok(output) => output,\n    Err(payload) => {\n        let msg = payload.downcast_ref::<String>().cloned()\n            .or_else(|| payload.downcast_ref::<&str>().map(|s| s.to_string()))\n            .unwrap_or_default();\n        eprintln!(\"DDP worker failed: {msg}\"); // includes the failing worker id\n        // degrade gracefully: retry on the main device only\n        learner.fit(devices[0].clone(), dataloader_train, dataloader_valid)\n    }\n}","preventionTips":["Verify every device index in the devices list exists and is idle/available before launching DDP","Make sure the dataset splits evenly so every peer's dataloader is non-empty","Run a quick single-device pass first to catch panics unrelated to distribution","Avoid panicking custom metrics/handlers that run inside worker threads","Keep backend drivers (GPU drivers, wgpu adapters, CUDA/torch libs) up to date and tested"],"tags":["distributed-training","panic","multi-device","ddp"],"backgroundTag":"worker-thread-panicked","analyzedSha":"d16f7ba2ed0d41408189384044cc886fb4c8f957","analyzedAt":"2026-09-05T13:19:14.260Z","contentChangedAt":"2026-09-05T13:19:14.260Z","schemaVersion":2},"datasetVersion":"2026-09-12T17:17:11.597Z"}