What causes "Can receive callback" in tracel-ai/burn?

Ensure every rank in the distributed job stays alive and participates in the collective sync — a single crashed peer will hang/panic the others.

tracel-ai/burn · critical

Can receive callback

Error message

Can receive callback

What it means

This panic occurs in submit_sync_collective after the caller sends a CollectiveSync request to the sync server and blocks on `rx.recv()` for the returned callback. The expect fires if the oneshot receiver returns Err, meaning the server never sent the closure back — typically because the server thread died or the callback sender was dropped without sending (see error 574's sibling panic).

Source

Thrown at crates/burn-backend/src/backend/distributed/client.rs:67

    pub fn submit_gradient_sync(&self, tensor: TensorRef<B>, params: DistributedParams) {
        self.sender
            .send(ActionMessage::Message(DistributedSyncMessage::TensorSync(
                (tensor, params),
            )))
            .unwrap();
    }

    pub fn submit_sync_collective(&self, device: Device<B>) {
        let (tx, rx) = oneshot::channel();

        self.sender
            .send(ActionMessage::Message(
                DistributedSyncMessage::CollectiveSync((device.clone(), tx)),
            ))
            .unwrap();

        let sync = rx.recv().expect("Can receive callback");

        sync();
    }

    pub(crate) fn close(&self) {
        self.sender.send(ActionMessage::Close()).unwrap();
    }
}

View on GitHub (pinned to d16f7ba2ed)

Solutions

Ensure every rank in the distributed job stays alive and participates in the collective sync — a single crashed peer will hang/panic the others.
Verify DistributedConfig (especially num_devices) matches the actual device set on all ranks.
Check logs on peer devices for panics (e.g. 574 'Can send callback') that starve this recv.
Add a timeout/multi-plex health check around collectives so dead peers surface as clear errors instead of recv Err panics.

Example fix

// before
let sync = rx.recv().expect("Can receive callback");
sync();
// after
let sync = rx.recv().map_err(|_|
    anyhow::anyhow!("gradient sync server dropped the callback; is a peer down?")
)?;
sync();

Defensive patterns

Strategy: retry

Try / catch

let sync = rx.recv().map_err(|_| {
    anyhow::anyhow!("sync server dropped callback; verify all ranks are alive")
})?;
sync();

Prevention

Monitor peer health before each collective; abort early if a rank is down.
Set num_devices in DistributedConfig to the true device count on every rank.
Check peer logs for panics that would starve this recv.

When it happens

Trigger: Calling submit_sync_collective when the background sync server thread is no longer alive (crashed or channel closed), or the server removed the device's callback and sent nothing due to a mismatch in device ids between sender and server.

Common situations: One rank in a distributed job panicked or exited, leaving other ranks waiting forever for a collective sync; mismatched num_devices in DistributedConfig causing the server never to reach the launch condition; running under a runtime where the spawned server thread failed to start.

Related errors

AI-assisted analysis of tracel-ai/burn@d16f7ba2ed (2026-09-05). Data as JSON: /api/errors/bd83b5c6d1c6a8a0. Report an issue: GitHub.