{"record":{"id":"632384ae44748c28","repo":"tracel-ai/burn","slug":"can-send-callback","errorCode":null,"errorMessage":"Can send callback","messagePattern":"Can send callback","errorType":"panic","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"crates/burn-backend/src/backend/distributed/server.rs","lineNumber":85,"sourceCode":"        self.launch_ops();\n    }\n\n    fn collective_sync(\n        &mut self,\n        device: Device<B>,\n        callback: oneshot::Sender<Box<dyn FnOnce() + Send>>,\n    ) {\n        self.callbacks.insert(device.id(), callback);\n        self.syncing_devices.push(device.clone());\n        self.try_launch_sync();\n    }\n\n    fn try_launch_sync(&mut self) {\n        if self.all_reduce_ops_queue.is_empty() {\n            for d in self.syncing_devices.clone() {\n                let callback = self.callbacks.remove(&d.id()).unwrap();\n                let closure = Box::new(move || B::sync_collective(&d));\n                callback.send(closure).expect(\"Can send callback\");\n                self.devices_synced += 1;\n            }\n            self.syncing_devices.clear();\n        }\n\n        if self.devices_synced == self.num_devices {\n            self.devices_registered = 0;\n            self.devices_synced = 0;\n            self.param_required_map.clear();\n            self.callbacks.clear();\n        }\n    }\n\n    fn launch_ops(&mut self) {\n        if self.devices_registered == self.num_devices {\n            for (param_id, num_tensors) in self.param_required_map.clone() {\n                let queued_tensors = self.all_reduce_ops_queue.entry(param_id).or_insert(vec![]);\n","sourceCodeStart":67,"sourceCodeEnd":103,"githubUrl":"https://github.com/tracel-ai/burn/blob/d16f7ba2ed0d41408189384044cc886fb4c8f957/crates/burn-backend/src/backend/distributed/server.rs#L67-L103","documentation":"This panic happens in try_launch_sync on the distributed sync server when it tries to send the sync closure back to a waiting device via a oneshot channel and `callback.send(...)` fails. send fails only when the receiver end was dropped — i.e. the device that requested the collective sync is gone before receiving its callback. It is called from collective_sync and launch_ops once all queued all-reduce ops are ready.","triggerScenarios":"A device calls submit_sync_collective, then its thread/task dies (panic, kill, dropped client) before the server dispatches the callback; the server's callbacks map contains a stale entry for a dead device id when try_launch_sync drains syncing_devices.","commonSituations":"Distributed training where one worker is OOM-killed mid-collective; timeouts that drop client tasks server-side; heterogeneous clusters where one rank runs out of memory during sync_collective and its task unwinds, dropping the oneshot receiver.","solutions":["Check the failing device for crashes (OOM, panic) that dropped the oneshot receiver before the callback arrived.","Ensure the calling task that owns the oneshot receiver is kept alive until sync() has run.","Add liveness/health handling in the server so dead devices are removed from syncing_devices instead of causing panics.","Confirm identical num_devices across all ranks so launch conditions are met simultaneously."],"exampleFix":"// before\nlet callback = self.callbacks.remove(&d.id()).unwrap();\ncallback.send(closure).expect(\"Can send callback\");\n// after\nif let Some(callback) = self.callbacks.remove(&d.id()) {\n    let _ = callback.send(closure); // receiver dropped: device is gone, skip it\n} else {\n    log::warn!(\"device {:?} disappeared during sync\", d.id());\n}","handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"if let Some(cb) = self.callbacks.remove(&d.id()) {\n    if cb.send(closure).is_err() {\n        log::warn!(\"device {:?} dropped before sync callback\", d.id());\n    }\n}","preventionTips":["Keep the task owning the oneshot receiver alive until sync() runs.","Remove dead devices from syncing_devices on disconnect instead of panicking.","Watch worker memory (OOM kills are a common cause of dropped receivers)."],"tags":["distributed","collective-sync","panic","oneshot-channel"],"backgroundTag":"collective-sync-peer-lost","analyzedSha":"d16f7ba2ed0d41408189384044cc886fb4c8f957","analyzedAt":"2026-09-05T13:19:14.260Z","contentChangedAt":"2026-09-05T13:19:14.260Z","schemaVersion":2},"datasetVersion":"2026-09-12T17:17:11.597Z"}