tracel-ai/burn · error

Gradient sync server disconnected.

Error message

Gradient sync server disconnected.

What it means

This is a panic from `.expect("Gradient sync server disconnected.")` on `rx.recv()` inside the background worker thread spawned by the distributed gradient-sync client. It fires when the mpsc channel's receiving side is dropped/closed while the server loop is still waiting for messages, i.e. the sync server thread lost its message source. It indicates the distributed synchronization infrastructure has broken down.

Source

Thrown at crates/burn-backend/src/backend/distributed/client.rs:34

    RegisterSyncParameters(Vec<DistributedParams>),
    TensorSync((TensorRef<B>, DistributedParams)),
    #[allow(clippy::type_complexity)]
    CollectiveSync((Device<B>, oneshot::Sender<Box<dyn FnOnce() + Send>>)),
}

#[derive(Clone)]
pub struct DistributedSyncClient<B: Backend> {
    sender: Sender<ActionMessage<B>>,
}

impl<B: Backend> DistributedSyncClient<B> {
    pub(crate) fn new(num_devices: usize, config: DistributedConfig) -> Self {
        let (tx, rx) = std::sync::mpsc::channel();

        let mut server = DistributedSyncServer::new(num_devices, config);
        spawn(move || {
            while let ActionMessage::Message(msg) =
                rx.recv().expect("Gradient sync server disconnected.")
            {
                server.process_message(msg)
            }
        });
        Self { sender: tx }
    }

    pub fn register_sync_parameters(&self, sharded_params: Vec<DistributedParams>) {
        self.sender
            .send(ActionMessage::Message(
                DistributedSyncMessage::RegisterSyncParameters(sharded_params),
            ))
            .unwrap();
    }

    pub fn submit_gradient_sync(&self, tensor: TensorRef<B>, params: DistributedParams) {
        self.sender
            .send(ActionMessage::Message(DistributedSyncMessage::TensorSync(

View on GitHub (pinned to d16f7ba2ed)

Solutions

1. Ensure graceful shutdown: send ActionMessage::Close() before dropping the channel (the client's close() does this).
2. Check teardown ordering so the server thread is joined/closed before the receiver is dropped.
3. Verify all participating devices use the same burn version and DistributedConfig so no side dies mid-run.
4. Inspect earlier panics in the worker thread — a preceding panic often causes the channel teardown.

Example fix

// before
// dropping client without closing
let client = GradientSyncClient::new(n, config);
drop(client);
// after
let client = GradientSyncClient::new(n, config);
client.close(); // sends ActionMessage::Close, then drop safely

// wrap worker loop so disconnect is reported, not panicked
match rx.recv() {
    Ok(ActionMessage::Message(msg)) => server.process_message(msg),
    Ok(ActionMessage::Close()) | Err(_) => break,
}

When it happens

Trigger: The receiver `rx` is dropped while the spawned thread is blocked in `rx.recv()`; the DistributedSyncServer-side channel is closed prematurely during teardown, or the client's channel pair is mismanaged so recv returns Err.

Common situations: Shutting down a distributed run where teardown order drops the receiver before the worker exits; a panic elsewhere in the worker causing the channel to be torn down; misconfigured distributed runs where device counts do not match and cleanup happens early.

Related errors

• Can receive callback
• Distributed operations are not supported for tensor kind {ot
• Distributed operations are not supported for device {other:?
• Autodiff should not wrap an autodiff device.
• an autodiff float primitive must have an enabled autodiff co

AI-assisted analysis of tracel-ai/burn@d16f7ba2ed (2026-09-05). Data as JSON: /api/errors/48d4adaa22bdcf4f. Report an issue: GitHub.