vllm-project/vllm · critical · BenchError

JSON error: {0}

Error message

JSON error: {0}

What it means

Raised by MoRIIOWrapper.register_local_tensor (moriio_engine.py:604) when the underlying MoRIIO engine fails to register a torch.Tensor as RDMA-accessible local memory, or when register_torch_tensor returns None. Registration pins/maps the tensor's storage so a remote node can RDMA read/write it; the resulting MemoryDesc is packed and exchanged with the peer to build the RDMA session. Any exception from the engine (invalid tensor, non-CUDA/non-pinned memory, OOM during registration, IBV registration failure) is wrapped in MoRIIOError with the cause chained via 'from e'.

Source

Thrown at rust/src/bench/src/error.rs:11

// SPDX-License-Identifier: Apache-2.0
// SPDX-FileCopyrightText: Copyright contributors to the vLLM project

use thiserror::Error;

#[derive(Error, Debug)]
pub enum BenchError {
    #[error("HTTP request failed: {0}")]
    Http(#[from] reqwest::Error),

    #[error("JSON error: {0}")]
    Json(#[from] serde_json::Error),

    #[error("Tokenizer error: {0}")]
    Tokenizer(String),

    /// The server's /tokenize//detokenize endpoint is not usable (4xx status:
    /// not exposed, or rejected by a gateway such as LLM-d/EPP that returns
    /// 400 instead of 404). Callers treat this as "skip verification", unlike
    /// `Tokenizer` errors which are genuine failures.
    #[error("tokenize endpoint unavailable: {0}")]
    TokenizeUnavailable(String),

    #[error("Configuration error: {0}")]
    Config(String),

    #[error("Endpoint not ready after {0}s: {1}")]
    EndpointTimeout(u64, String),

View on GitHub (pinned to c794754062)

Solutions

1. Read the chained cause: the '{e}' text names the real failure — act on that message first (e.g. 'failed to register memory region', 'invalid device', 'None returned').
2. Verify the tensor being registered is the real CUDA KV cache tensor: .is_cuda, .is_contiguous(), .numel() > 0, and lives on the device the MoRIIO engine was initialized with.
3. Check RDMA stack health on the node: the NIC is up, its GID routes to the GPU (nvidia-smi topo -m, ibstat), and CUDA_VISIBLE_DEVICES does not remap device indices unexpectedly.
4. If registration fails with MR/resource errors, restart the engine process (leaked registrations from earlier attempts hold MRs) and reduce number of registered tensors.
5. Upgrade/align the MoRIIO package version so register_torch_tensor either raises a descriptive error or never returns None.

Example fix

// before
packed = wrapper.register_local_tensor(kv_cache)  # may wrap opaque engine errors

// after (validate eligibility first, then surface the chained cause)
assert kv_cache.is_cuda and kv_cache.is_contiguous() and kv_cache.numel() > 0, (
    f"kv_cache not registrable: cuda={kv_cache.is_cuda} "
    f"contiguous={kv_cache.is_contiguous()} numel={kv_cache.numel()}"
)
try:
    packed = wrapper.register_local_tensor(kv_cache)
except MoRIIOError as e:
    raise RuntimeError(
        f"RDMA registration of KV cache failed on device "
        f"{kv_cache.device}: {e.__cause__ or e}"
    ) from e

def is_registrable_kv_cache(tensor) -> bool:
    return (
        tensor.is_cuda
        and tensor.is_contiguous()
        and tensor.numel() > 0
        and tensor.data_ptr() != 0
    )

# before engine startup / registration:
assert is_registrable_kv_cache(kv_cache), (
    f"KV cache not RDMA-registrable: cuda={kv_cache.is_cuda}, "
    f"contig={kv_cache.is_contiguous()}, numel={kv_cache.numel()}"
)
wrapper.register_local_tensor(kv_cache)

from typing import TypeGuard
import torch

def is_cuda_kv_cache_tensor(t: object) -> TypeGuard[torch.Tensor]:
    return (
        isinstance(t, torch.Tensor)
        and t.is_cuda
        and t.is_contiguous()
        and t.numel() > 0
        and not t.requires_grad
    )

from vllm.distributed.kv_transfer.kv_connector.v1.moriio.moriio_engine import MoRIIOError

try:
    packed_meta = wrapper.register_local_tensor(kv_cache)
except MoRIIOError as e:
    cause = e.__cause__ or e
    logger.error("RDMA registration failed on %s: %s", kv_cache.device, cause)
    raise RuntimeError(
        "KV cache RDMA registration failed; check GPU/NIC affinity "
        "(nvidia-smi topo -m) and MoRIIO engine init"
    ) from e

When it happens

Trigger: Calling register_local_tensor(kv_cache) as the connector does at moriio_connector.py:1775 during KV-role startup, where the tensor is not eligible for RDMA registration: CPU tensor that is not pinned, a non-contiguous or zero-element tensor, a view whose storage the NIC cannot register, CUDA memory on a device invisible to the RDMA stack, or ibv_reg_mr failure from an IOMMU/vfio mismatch or MR limit exhaustion. Also triggered when register_torch_tensor returns None, which the assert converts into the same MoRIIOError.

Common situations: Running MoRIIO disaggregated prefill/decode on nodes where the GPU is not on the same NUMA/IOMMU domain as the RDMA NIC; CUDA_VISIBLE_DEVICES hiding the GPU the NIC's GID refers to; kv cache tensors created with torch.empty on CPU without page-locked memory in CPU-offload setups; older MoRIIO builds whose register_torch_tensor returns None instead of raising; exceeding the NIC's memory-region limit after many engine restarts in one process.

Related errors

• Tokenizer error: {0}
• HTTP request failed: {0}
• tokenize endpoint unavailable: {0}
• Mooncake batch memory registration failed.
• Invalid MoRIIO backend {backend!r} in kv_connector_extra_con

AI-assisted analysis of vllm-project/vllm@c794754062 (2026-08-14). Data as JSON: /api/errors/c2d1e2f6579bf35d. Report an issue: GitHub.