What causes "Failed to read a tensor off {from:?} on the way to {to:?}: {err}" in tracel-ai/burn?

Read the inner `{err}` for the concrete read failure from the source runtime

Failed to read a tensor off {from:?} on the way to {to:?}: {

Error message

Failed to read a tensor off {from:?} on the way to {to:?}: {err}

What it means

transfer_failed panics when reading a tensor's data off one CubeCL device fails while moving it to another runtime/device (to_device_across_runtimes). The message names the source device, the destination device, and the underlying read error. Cross-runtime transfer requires reading the tensor to host memory, so a failed read blocks the whole migration.

Source

Thrown at crates/burn-cubecl/src/ops/base.rs:129

        .client
        .read_one(tensor.handle.clone())
        .unwrap_or_else(|err| transfer_failed(&tensor.device, device, err));

    let client = device.client();
    let handle = client.create(bytes);

    CubeTensor {
        client,
        handle,
        meta: tensor.meta,
        device: device.clone(),
        dtype: tensor.dtype,
        qparams: tensor.qparams,
    }
}

fn transfer_failed(from: &CubeDevice, to: &CubeDevice, err: impl core::fmt::Display) -> ! {
    panic!("Failed to read a tensor off {from:?} on the way to {to:?}: {err}")
}

pub(crate) fn empty(shape: Shape, device: &CubeDevice, dtype: DType) -> CubeTensor {
    let client = device.client();
    let alloc = client.empty_tensor(shape.clone(), dtype.size());

    CubeTensor::new(
        client,
        alloc.memory,
        Metadata::new(shape, alloc.strides),
        device.clone(),
        dtype,
    )
}

pub(crate) fn swap_dims(mut tensor: CubeTensor, dim1: usize, dim2: usize) -> CubeTensor {
    tensor.meta.swap(dim1, dim2);

View on GitHub (pinned to d16f7ba2ed)

Solutions

Read the inner `{err}` for the concrete read failure from the source runtime
Verify the source device/client is still alive and the runtime is healthy
Re-create the tensor on the source device if the handle became invalid, then transfer
As a workaround, read the tensor to host with an explicit to_data() and re-send it to the target device

Example fix

// before
let on_wgpu = tensor.to_device_across_runtimes(&wgpu_device);
// after: host round-trip fallback
let data = tensor.into_data().unwrap();
let on_wgpu = Tensor::from_data(data, &wgpu_device);

Defensive patterns

Strategy: try-catch

Validate before calling

// verify both runtimes are alive before transfer
assert!(source_client.is_valid(), "source runtime unavailable");

Try / catch

match std::panic::catch_unwind(|| tensor.to_device_across_runtimes(&target)) {
    Ok(t) => t,
    Err(_) => {
        let data = tensor.into_data().expect("host read");
        Tensor::from_data(data, &target)
    }
}

Prevention

Prefer host round-trip (into_data + from_data) when crossing runtimes
Avoid holding tensors on a client you're about to drop
Test mixed-backend transfers once at startup, not mid-training

When it happens

Trigger: Calling to_device_across_runtimes (e.g. moving a tensor from a CUDA client to a Vulkan/Metal/wgpu client) when the source runtime errors while reading tensor data — invalid handle, runtime driver error, or the tensor was freed.

Common situations: Mixed-backend setups (cuda + wgpu) where the source runtime crashed or the driver misbehaves, transferring tensors after the source client was dropped, or dtype/shape mismatch causing an invalid read size.

Related errors

AI-assisted analysis of tracel-ai/burn@d16f7ba2ed (2026-09-05). Data as JSON: /api/errors/cace1e2353f52330. Report an issue: GitHub.