tracel-ai/burn · error

not implemented

Error message

not implemented

What it means

burn-dispatch's distributed backend implements `submit_gradient_sync` as `unimplemented!()`. Unlike the other DistributedOps methods, it does not forward to the inner backend `B`, so any call always panics with 'not implemented'. The library authors have not wired gradient synchronization through the dispatch layer yet.

Source

Thrown at crates/burn-dispatch/src/ops/distributed.rs:209

        })
    }

    fn register_sync_parameters(
        device: &DispatchDevice,
        sharded_param_ids: Vec<DistributedParams>,
    ) {
        dispatch_device!(@distributed device, |device| B::register_sync_parameters(
            device,
            sharded_param_ids,
        ))
    }

    fn submit_sync_collective(device: &DispatchDevice) {
        dispatch_device!(@distributed device, |device| B::submit_sync_collective(device))
    }

    fn submit_gradient_sync(_tensor: TensorRef<Self>, _distributed_params: DistributedParams) {
        unimplemented!()
    }

    fn all_reduce(
        tensor: FloatTensor<Self>,
        op: ReduceOperation,
        device_ids: Vec<DeviceId>,
    ) -> CollectiveTensor<Self> {
        // Safety: we call `assume_resolved` only to wrap it in a new `CollectiveTensor`.
        // Explicit type: the distributed dispatch only emits arms for collective-capable
        // backends (Cuda, Remote), so a build with none of them leaves only the diverging
        // fallback and the match would otherwise infer `!`.
        let tensor: FloatTensor<Self> = dispatch_distributed_float!(tensor, |tensor| {
            let collective_tensor = B::all_reduce(tensor, op, device_ids);
            unsafe { collective_tensor.assume_resolved() }
        });
        CollectiveTensor::new(tensor)
    }

View on GitHub (pinned to d16f7ba2ed)

Solutions

1. Use a concrete distributed backend for gradient sync instead of the Dispatch backend
2. Check the burn repo for an updated release where Dispatch forwards this op to the inner backend
3. Contribute/implement the op by forwarding like the sibling `submit_sync_collective`: `dispatch_device!(@distributed device, |device| B::submit_gradient_sync(...))`
4. Work around by performing gradient sync manually outside the dispatch layer

Example fix

// before
fn submit_gradient_sync(_tensor: TensorRef<Self>, _distributed_params: DistributedParams) {
    unimplemented!()
}
// after
fn submit_gradient_sync(tensor: TensorRef<Self>, distributed_params: DistributedParams) {
    dispatch_device!(@distributed ..., |device| B::submit_gradient_sync(tensor, distributed_params))
}

if backend_is_dispatch() && needs_gradient_sync() {
    panic/branch to a real distributed backend before calling submit_gradient_sync
}

When it happens

Trigger: Calling `submit_gradient_sync(tensor, distributed_params)` on the Dispatch backend, e.g. through a distributed training loop that relies on the DistributedOps trait to sync gradients across devices.

Common situations: Running multi-device/multi-node distributed training while routing tensors through burn-dispatch instead of a concrete distributed backend; copying code that works against a real backend (e.g. NCCL-based) to the dispatch layer.

Related errors

• ctc_loss: 2 * max_target_len + 1 = {} exceeds the kernel's s
• ctc_loss_backward: 2 * max_target_len + 1 = {} exceeds the k
• Node {:?} is needed but never checkpointed
• Trying to consume the gradients for an untracked tensor
• Invalid broadcast shapes: Next grad shape {:?}, Previous gra

AI-assisted analysis of tracel-ai/burn@d16f7ba2ed (2026-09-05). Data as JSON: /api/errors/361cc3f76c672189. Report an issue: GitHub.