tracel-ai/burn · critical

Can receive callback

Error message

Can receive callback

What it means

This panic occurs in submit_sync_collective after the caller sends a CollectiveSync request to the sync server and blocks on `rx.recv()` for the returned callback. The expect fires if the oneshot receiver returns Err, meaning the server never sent the closure back — typically because the server thread died or the callback sender was dropped without sending (see error 574's sibling panic).

Solutions

1. Ensure every rank in the distributed job stays alive and participates in the collective sync — a single crashed peer will hang/panic the others.
2. Verify DistributedConfig (especially num_devices) matches the actual device set on all ranks.
3. Check logs on peer devices for panics (e.g. 574 'Can send callback') that starve this recv.
4. Add a timeout/multi-plex health check around collectives so dead peers surface as clear errors instead of recv Err panics.

Example fix

// before
let sync = rx.recv().expect("Can receive callback");
sync();
// after
let sync = rx.recv().map_err(|_|
    anyhow::anyhow!("gradient sync server dropped the callback; is a peer down?")
)?;
sync();

let sync = rx.recv().map_err(|_| {
    anyhow::anyhow!("sync server dropped callback; verify all ranks are alive")
})?;
sync();

When it happens

Trigger: Calling submit_sync_collective when the background sync server thread is no longer alive (crashed or channel closed), or the server removed the device's callback and sent nothing due to a mismatch in device ids between sender and server.

Common situations: One rank in a distributed job panicked or exited, leaving other ranks waiting forever for a collective sync; mismatched num_devices in DistributedConfig causing the server never to reach the launch condition; running under a runtime where the spawned server thread failed to start.

Related errors

• Can send callback
• Gradient sync server disconnected.
• an autodiff float primitive must have an enabled autodiff…
• Autodiff should not wrap an autodiff device.
• Can't register manually a tensor on a remote channel.

AI-assisted analysis of tracel-ai/burn@d16f7ba2ed (2026-09-05). Data as JSON: /api/errors/bd83b5c6d1c6a8a0. Report an issue: GitHub.

Appendix: source

Thrown at crates/burn-backend/src/backend/distributed/client.rs:67

    pub fn submit_gradient_sync(&self, tensor: TensorRef<B>, params: DistributedParams) {
        self.sender
            .send(ActionMessage::Message(DistributedSyncMessage::TensorSync(
                (tensor, params),
            )))
            .unwrap();
    }

    pub fn submit_sync_collective(&self, device: Device<B>) {
        let (tx, rx) = oneshot::channel();

        self.sender
            .send(ActionMessage::Message(
                DistributedSyncMessage::CollectiveSync((device.clone(), tx)),
            ))
            .unwrap();

        let sync = rx.recv().expect("Can receive callback");

        sync();
    }

    pub(crate) fn close(&self) {
        self.sender.send(ActionMessage::Close()).unwrap();
    }
}

View on GitHub (pinned to d16f7ba2ed)