{"record":{"id":"2dd6ca2ae6e53557","repo":"tracel-ai/burn","slug":"training-worker-on-device-device-id-failed-msg","errorCode":null,"errorMessage":"training worker on device {device_id} failed: {msg}","messagePattern":"training worker on device (.+?) failed: (.+?)","errorType":"panic","errorClass":null,"httpStatus":null,"severity":"error","filePath":"crates/burn-train/src/learner/supervised/step/train.rs","lineNumber":182,"sourceCode":"                Some(Ok(item)) => {\n                    worker.register(item, model);\n                    num_send += 1;\n                    let progress = dataloader.progress();\n                    items_total += progress.items_total;\n                    items_processed += progress.items_processed;\n                }\n                Some(Err(err)) => return Err(err),\n                None => {}\n            }\n        }\n\n        let mut outputs = Vec::with_capacity(num_send);\n\n        for _ in 0..num_send {\n            match self.receiver.recv().unwrap() {\n                WorkerMessage::Output(output) => outputs.push(output),\n                WorkerMessage::Error(device_id, msg) => {\n                    panic!(\"training worker on device {device_id} failed: {msg}\");\n                }\n            }\n        }\n\n        Ok((outputs, Progress::new(items_processed, items_total, unit)))\n    }\n}\n","sourceCodeStart":164,"sourceCodeEnd":190,"githubUrl":"https://github.com/tracel-ai/burn/blob/d16f7ba2ed0d41408189384044cc886fb4c8f957/crates/burn-train/src/learner/supervised/step/train.rs#L164-L190","documentation":"The distributed/multi-device training step aborts because a training worker on the given device returned an error message `msg` (e.g., the model forward/backward failed or the worker channel closed). The step cannot produce outputs for the batch, so the error from the worker is propagated to the caller.","triggerScenarios":"Thrown at crates/burn-train/src/learner/supervised/step/train.rs:182 when the library encounters an invalid state.","commonSituations":"See trigger scenarios.","solutions":["Read `msg` for the underlying worker failure (OOM, device fault, shape mismatch)","Reduce batch size if the worker ran out of device memory","Verify the device is healthy and accessible","Ensure the dataloader and model run correctly on that device in isolation"],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":null,"preventionTips":[],"tags":[],"backgroundTag":null,"analyzedSha":"d16f7ba2ed0d41408189384044cc886fb4c8f957","analyzedAt":"2026-09-05T13:19:14.260Z","contentChangedAt":"2026-09-05T13:19:14.260Z","schemaVersion":2},"datasetVersion":"2026-09-12T17:17:11.597Z"}