What causes "Can send callback" in tracel-ai/burn?

Check the failing device for crashes (OOM, panic) that dropped the oneshot receiver before the callback arrived.

Can send callback

Error message

Can send callback

What it means

This panic happens in try_launch_sync on the distributed sync server when it tries to send the sync closure back to a waiting device via a oneshot channel and `callback.send(...)` fails. send fails only when the receiver end was dropped — i.e. the device that requested the collective sync is gone before receiving its callback. It is called from collective_sync and launch_ops once all queued all-reduce ops are ready.

Source

Thrown at crates/burn-backend/src/backend/distributed/server.rs:85

        self.launch_ops();
    }

    fn collective_sync(
        &mut self,
        device: Device<B>,
        callback: oneshot::Sender<Box<dyn FnOnce() + Send>>,
    ) {
        self.callbacks.insert(device.id(), callback);
        self.syncing_devices.push(device.clone());
        self.try_launch_sync();
    }

    fn try_launch_sync(&mut self) {
        if self.all_reduce_ops_queue.is_empty() {
            for d in self.syncing_devices.clone() {
                let callback = self.callbacks.remove(&d.id()).unwrap();
                let closure = Box::new(move || B::sync_collective(&d));
                callback.send(closure).expect("Can send callback");
                self.devices_synced += 1;
            }
            self.syncing_devices.clear();
        }

        if self.devices_synced == self.num_devices {
            self.devices_registered = 0;
            self.devices_synced = 0;
            self.param_required_map.clear();
            self.callbacks.clear();
        }
    }

    fn launch_ops(&mut self) {
        if self.devices_registered == self.num_devices {
            for (param_id, num_tensors) in self.param_required_map.clone() {
                let queued_tensors = self.all_reduce_ops_queue.entry(param_id).or_insert(vec![]);

View on GitHub (pinned to d16f7ba2ed)

Solutions

Check the failing device for crashes (OOM, panic) that dropped the oneshot receiver before the callback arrived.
Ensure the calling task that owns the oneshot receiver is kept alive until sync() has run.
Add liveness/health handling in the server so dead devices are removed from syncing_devices instead of causing panics.
Confirm identical num_devices across all ranks so launch conditions are met simultaneously.

Example fix

// before
let callback = self.callbacks.remove(&d.id()).unwrap();
callback.send(closure).expect("Can send callback");
// after
if let Some(callback) = self.callbacks.remove(&d.id()) {
    let _ = callback.send(closure); // receiver dropped: device is gone, skip it
} else {
    log::warn!("device {:?} disappeared during sync", d.id());
}

Defensive patterns

Strategy: try-catch

Try / catch

if let Some(cb) = self.callbacks.remove(&d.id()) {
    if cb.send(closure).is_err() {
        log::warn!("device {:?} dropped before sync callback", d.id());
    }
}

Prevention

Keep the task owning the oneshot receiver alive until sync() runs.
Remove dead devices from syncing_devices on disconnect instead of panicking.
Watch worker memory (OOM kills are a common cause of dropped receivers).

When it happens

Trigger: A device calls submit_sync_collective, then its thread/task dies (panic, kill, dropped client) before the server dispatches the callback; the server's callbacks map contains a stale entry for a dead device id when try_launch_sync drains syncing_devices.

Common situations: Distributed training where one worker is OOM-killed mid-collective; timeouts that drop client tasks server-side; heterogeneous clusters where one rank runs out of memory during sync_collective and its task unwinds, dropping the oneshot receiver.

Related errors

AI-assisted analysis of tracel-ai/burn@d16f7ba2ed (2026-09-05). Data as JSON: /api/errors/632384ae44748c28. Report an issue: GitHub.