{"record":{"id":"bd83b5c6d1c6a8a0","repo":"tracel-ai/burn","slug":"can-receive-callback","errorCode":null,"errorMessage":"Can receive callback","messagePattern":"Can receive callback","errorType":"panic","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"crates/burn-backend/src/backend/distributed/client.rs","lineNumber":67,"sourceCode":"\n    pub fn submit_gradient_sync(&self, tensor: TensorRef<B>, params: DistributedParams) {\n        self.sender\n            .send(ActionMessage::Message(DistributedSyncMessage::TensorSync(\n                (tensor, params),\n            )))\n            .unwrap();\n    }\n\n    pub fn submit_sync_collective(&self, device: Device<B>) {\n        let (tx, rx) = oneshot::channel();\n\n        self.sender\n            .send(ActionMessage::Message(\n                DistributedSyncMessage::CollectiveSync((device.clone(), tx)),\n            ))\n            .unwrap();\n\n        let sync = rx.recv().expect(\"Can receive callback\");\n\n        sync();\n    }\n\n    pub(crate) fn close(&self) {\n        self.sender.send(ActionMessage::Close()).unwrap();\n    }\n}\n","sourceCodeStart":49,"sourceCodeEnd":76,"githubUrl":"https://github.com/tracel-ai/burn/blob/d16f7ba2ed0d41408189384044cc886fb4c8f957/crates/burn-backend/src/backend/distributed/client.rs#L49-L76","documentation":"This panic occurs in submit_sync_collective after the caller sends a CollectiveSync request to the sync server and blocks on `rx.recv()` for the returned callback. The expect fires if the oneshot receiver returns Err, meaning the server never sent the closure back — typically because the server thread died or the callback sender was dropped without sending (see error 574's sibling panic).","triggerScenarios":"Calling submit_sync_collective when the background sync server thread is no longer alive (crashed or channel closed), or the server removed the device's callback and sent nothing due to a mismatch in device ids between sender and server.","commonSituations":"One rank in a distributed job panicked or exited, leaving other ranks waiting forever for a collective sync; mismatched num_devices in DistributedConfig causing the server never to reach the launch condition; running under a runtime where the spawned server thread failed to start.","solutions":["Ensure every rank in the distributed job stays alive and participates in the collective sync — a single crashed peer will hang/panic the others.","Verify DistributedConfig (especially num_devices) matches the actual device set on all ranks.","Check logs on peer devices for panics (e.g. 574 'Can send callback') that starve this recv.","Add a timeout/multi-plex health check around collectives so dead peers surface as clear errors instead of recv Err panics."],"exampleFix":"// before\nlet sync = rx.recv().expect(\"Can receive callback\");\nsync();\n// after\nlet sync = rx.recv().map_err(|_|\n    anyhow::anyhow!(\"gradient sync server dropped the callback; is a peer down?\")\n)?;\nsync();","handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"let sync = rx.recv().map_err(|_| {\n    anyhow::anyhow!(\"sync server dropped callback; verify all ranks are alive\")\n})?;\nsync();","preventionTips":["Monitor peer health before each collective; abort early if a rank is down.","Set num_devices in DistributedConfig to the true device count on every rank.","Check peer logs for panics that would starve this recv."],"tags":["distributed","collective-sync","panic","channels"],"backgroundTag":"collective-sync-peer-lost","analyzedSha":"d16f7ba2ed0d41408189384044cc886fb4c8f957","analyzedAt":"2026-09-05T13:19:14.260Z","contentChangedAt":"2026-09-05T13:19:14.260Z","schemaVersion":2},"datasetVersion":"2026-09-12T17:17:11.597Z"}