vllm-project/vllm · error · ValueError

Flashinfer allreduce is not supported for multi-node allredu

Error message

Flashinfer allreduce is not supported for multi-node allreduce with 'trtllm' backend. Please use 'mnnvl' backend instead.

What it means

get_flashinfer_allreduce_workspace builds the FlashInfer allreduce workspace for the non-quant fusion path. The backend is chosen from VLLM_FLASHINFER_ALLREDUCE_BACKEND (or 'auto'). The 'trtllm' allreduce kernel only works within a single node, so the guard raises ValueError when get_node_count() > 1 and the resolved backend is 'trtllm'; multi-node requires the 'mnnvl' multicast backend.

Source

Thrown at vllm/distributed/device_communicators/flashinfer_all_reduce.py:163

    hidden_dim: int,
    dtype: torch.dtype,
    group: ProcessGroup,
):
    """
    Return the allreduce workspace for non-quant patterns, initializing if needed.

    Used by AllReduceFusionPass (non-quant patterns) and FlashInferAllReduce
    for standalone allreduce. Backend is controlled by
    VLLM_FLASHINFER_ALLREDUCE_BACKEND env var.
    """
    global _fi_ar_workspace
    if _fi_ar_workspace is not None:
        return _fi_ar_workspace

    backend, allow_trtllm_fallback = _resolve_fi_ar_backend()

    if get_node_count() > 1 and backend == "trtllm":
        raise ValueError(
            "Flashinfer allreduce is not supported for multi-node allreduce with "
            "'trtllm' backend. Please use 'mnnvl' backend instead."
        )

    def _get_or_create(be: str):
        # Reuse the quant workspace if it was already created with the same backend
        if _fi_ar_quant_workspace is not None and _fi_ar_quant_workspace.backend == be:
            return _fi_ar_quant_workspace
        return _create_workspace(
            be, world_size, rank, max_token_num, hidden_dim, dtype, group
        )

    _fi_ar_workspace = _get_or_create(backend)
    if _fi_ar_workspace is None and allow_trtllm_fallback and backend != "trtllm":
        logger.warning_once(
            "FlashInfer mnnvl allreduce workspace unavailable (likely no NVSwitch "
            "multicast support); falling back to trtllm backend for single node."
        )

View on GitHub (pinned to c794754062)

Solutions

1. Set VLLM_FLASHINFER_ALLREDUCE_BACKEND=mnnvl for the multi-node deployment
2. If nodes lack multicast NVLink, disable FlashInfer allreduce fusion and fall back to the default PyNccl allreduce path
3. Restrict the trtllm backend to single-node topologies only

Example fix

# before
export VLLM_FLASHINFER_ALLREDUCE_BACKEND=trtllm
# multi-node launch -> ValueError

# after
export VLLM_FLASHINFER_ALLREDUCE_BACKEND=mnnvl
# or unset + disable fusion to use default allreduce

import os
from vllm.distributed import get_node_count
backend = os.environ.get("VLLM_FLASHINFER_ALLREDUCE_BACKEND", "auto")
if get_node_count() > 1:
    assert backend in ("mnnvl", "auto"), "multi-node flashinfer allreduce requires mnnvl backend"

When it happens

Trigger: Running a multi-node vLLM deployment (tensor parallel across nodes, get_node_count() > 1) with VLLM_FLASHINFER_ALLREDUCE_BACKEND=trtllm explicitly set (or auto-resolved to trtllm) while a graph with allreduce fusion is prepared, which calls get_flashinfer_allreduce_workspace.

Common situations: Copying a single-node config (which pinned trtllm) to a multi-node Ray/slurm deployment; enabling --enable-flashinfer-allreduce-fusion on nodes without NVLink multicast so 'auto' falls back to trtllm; upgrading to a topology where the previous backend choice no longer applies.

Related errors

• Flashinfer allreduce quantization fusion is not supported fo
• Unknown all2all backend: {self.all2all_backend}
• NCCL error: {error_str}
• VLLM_ROCM_QUICK_REDUCE_MIN_SIZE_BYTES_MB must be non-negativ
• VLLM_ROCM_QUICK_REDUCE_MIN_SIZE_BYTES_MB must be less than o

AI-assisted analysis of vllm-project/vllm@c794754062 (2026-08-14). Data as JSON: /api/errors/e1c3b65bb0eeb544. Report an issue: GitHub.