{"record":{"id":"48d4adaa22bdcf4f","repo":"tracel-ai/burn","slug":"gradient-sync-server-disconnected","errorCode":null,"errorMessage":"Gradient sync server disconnected.","messagePattern":"Gradient sync server disconnected\\.","errorType":"panic","errorClass":null,"httpStatus":null,"severity":"error","filePath":"crates/burn-backend/src/backend/distributed/client.rs","lineNumber":34,"sourceCode":"    RegisterSyncParameters(Vec<DistributedParams>),\n    TensorSync((TensorRef<B>, DistributedParams)),\n    #[allow(clippy::type_complexity)]\n    CollectiveSync((Device<B>, oneshot::Sender<Box<dyn FnOnce() + Send>>)),\n}\n\n#[derive(Clone)]\npub struct DistributedSyncClient<B: Backend> {\n    sender: Sender<ActionMessage<B>>,\n}\n\nimpl<B: Backend> DistributedSyncClient<B> {\n    pub(crate) fn new(num_devices: usize, config: DistributedConfig) -> Self {\n        let (tx, rx) = std::sync::mpsc::channel();\n\n        let mut server = DistributedSyncServer::new(num_devices, config);\n        spawn(move || {\n            while let ActionMessage::Message(msg) =\n                rx.recv().expect(\"Gradient sync server disconnected.\")\n            {\n                server.process_message(msg)\n            }\n        });\n        Self { sender: tx }\n    }\n\n    pub fn register_sync_parameters(&self, sharded_params: Vec<DistributedParams>) {\n        self.sender\n            .send(ActionMessage::Message(\n                DistributedSyncMessage::RegisterSyncParameters(sharded_params),\n            ))\n            .unwrap();\n    }\n\n    pub fn submit_gradient_sync(&self, tensor: TensorRef<B>, params: DistributedParams) {\n        self.sender\n            .send(ActionMessage::Message(DistributedSyncMessage::TensorSync(","sourceCodeStart":16,"sourceCodeEnd":52,"githubUrl":"https://github.com/tracel-ai/burn/blob/d16f7ba2ed0d41408189384044cc886fb4c8f957/crates/burn-backend/src/backend/distributed/client.rs#L16-L52","documentation":"This is a panic from `.expect(\"Gradient sync server disconnected.\")` on `rx.recv()` inside the background worker thread spawned by the distributed gradient-sync client. It fires when the mpsc channel's receiving side is dropped/closed while the server loop is still waiting for messages, i.e. the sync server thread lost its message source. It indicates the distributed synchronization infrastructure has broken down.","triggerScenarios":"The receiver `rx` is dropped while the spawned thread is blocked in `rx.recv()`; the DistributedSyncServer-side channel is closed prematurely during teardown, or the client's channel pair is mismanaged so recv returns Err.","commonSituations":"Shutting down a distributed run where teardown order drops the receiver before the worker exits; a panic elsewhere in the worker causing the channel to be torn down; misconfigured distributed runs where device counts do not match and cleanup happens early.","solutions":["Ensure graceful shutdown: send ActionMessage::Close() before dropping the channel (the client's close() does this).","Check teardown ordering so the server thread is joined/closed before the receiver is dropped.","Verify all participating devices use the same burn version and DistributedConfig so no side dies mid-run.","Inspect earlier panics in the worker thread — a preceding panic often causes the channel teardown."],"exampleFix":"// before\n// dropping client without closing\nlet client = GradientSyncClient::new(n, config);\ndrop(client);\n// after\nlet client = GradientSyncClient::new(n, config);\nclient.close(); // sends ActionMessage::Close, then drop safely","handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"// wrap worker loop so disconnect is reported, not panicked\nmatch rx.recv() {\n    Ok(ActionMessage::Message(msg)) => server.process_message(msg),\n    Ok(ActionMessage::Close()) | Err(_) => break,\n}","preventionTips":["Always call client.close() before dropping the sync client.","Keep all ranks alive for the duration of the distributed run.","Match burn versions and DistributedConfig across all nodes."],"tags":["distributed","channels","panic","shutdown"],"backgroundTag":"channel-disconnected-panic","analyzedSha":"d16f7ba2ed0d41408189384044cc886fb4c8f957","analyzedAt":"2026-09-05T13:19:14.260Z","contentChangedAt":"2026-09-05T13:19:14.260Z","schemaVersion":2},"datasetVersion":"2026-09-12T17:17:11.597Z"}