tracel-ai/burn · error

Gradient sync server disconnected.

Error message

Gradient sync server disconnected.

What it means

This is a panic from `.expect("Gradient sync server disconnected.")` on `rx.recv()` inside the background worker thread spawned by the distributed gradient-sync client. It fires when the mpsc channel's receiving side is dropped/closed while the server loop is still waiting for messages, i.e. the sync server thread lost its message source. It indicates the distributed synchronization infrastructure has broken down.

Solutions

1. Ensure graceful shutdown: send ActionMessage::Close() before dropping the channel (the client's close() does this).
2. Check teardown ordering so the server thread is joined/closed before the receiver is dropped.
3. Verify all participating devices use the same burn version and DistributedConfig so no side dies mid-run.
4. Inspect earlier panics in the worker thread — a preceding panic often causes the channel teardown.

Example fix

// before
// dropping client without closing
let client = GradientSyncClient::new(n, config);
drop(client);
// after
let client = GradientSyncClient::new(n, config);
client.close(); // sends ActionMessage::Close, then drop safely

// wrap worker loop so disconnect is reported, not panicked
match rx.recv() {
    Ok(ActionMessage::Message(msg)) => server.process_message(msg),
    Ok(ActionMessage::Close()) | Err(_) => break,
}

When it happens

Trigger: The receiver `rx` is dropped while the spawned thread is blocked in `rx.recv()`; the DistributedSyncServer-side channel is closed prematurely during teardown, or the client's channel pair is mismanaged so recv returns Err.

Common situations: Shutting down a distributed run where teardown order drops the receiver before the worker exits; a panic elsewhere in the worker causing the channel to be torn down; misconfigured distributed runs where device counts do not match and cleanup happens early.

Related errors

• Can receive callback
• an autodiff float primitive must have an enabled autodiff…
• Autodiff should not wrap an autodiff device.
• Can send callback
• Can't register manually a tensor on a remote channel.

AI-assisted analysis of tracel-ai/burn@d16f7ba2ed (2026-09-05). Data as JSON: /api/errors/48d4adaa22bdcf4f. Report an issue: GitHub.

Appendix: source

Thrown at crates/burn-backend/src/backend/distributed/client.rs:34

    RegisterSyncParameters(Vec<DistributedParams>),
    TensorSync((TensorRef<B>, DistributedParams)),
    #[allow(clippy::type_complexity)]
    CollectiveSync((Device<B>, oneshot::Sender<Box<dyn FnOnce() + Send>>)),
}

#[derive(Clone)]
pub struct DistributedSyncClient<B: Backend> {
    sender: Sender<ActionMessage<B>>,
}

impl<B: Backend> DistributedSyncClient<B> {
    pub(crate) fn new(num_devices: usize, config: DistributedConfig) -> Self {
        let (tx, rx) = std::sync::mpsc::channel();

        let mut server = DistributedSyncServer::new(num_devices, config);
        spawn(move || {
            while let ActionMessage::Message(msg) =
                rx.recv().expect("Gradient sync server disconnected.")
            {
                server.process_message(msg)
            }
        });
        Self { sender: tx }
    }

    pub fn register_sync_parameters(&self, sharded_params: Vec<DistributedParams>) {
        self.sender
            .send(ActionMessage::Message(
                DistributedSyncMessage::RegisterSyncParameters(sharded_params),
            ))
            .unwrap();
    }

    pub fn submit_gradient_sync(&self, tensor: TensorRef<B>, params: DistributedParams) {
        self.sender
            .send(ActionMessage::Message(DistributedSyncMessage::TensorSync(

View on GitHub (pinned to d16f7ba2ed)