vllm-project/vllm · error · ValueError

Flashinfer allreduce quantization fusion is not supported fo

Error message

Flashinfer allreduce quantization fusion is not supported for multi-node allreduce with 'trtllm' backend. Please use 'mnnvl' backend instead.

What it means

get_flashinfer_allreduce_quant_workspace is the quantization-fusion variant of the FlashInfer allreduce workspace (allreduce + per-tensor quant fused). It enforces the same topology constraint as the non-quant path: the 'trtllm' backend cannot drive cross-node allreduce, so backend == 'trtllm' with get_node_count() > 1 raises ValueError and directs you to 'mnnvl'.

Source

Thrown at vllm/distributed/device_communicators/flashinfer_all_reduce.py:221

    hidden_dim: int,
    dtype: torch.dtype,
    group: ProcessGroup,
):
    """
    Return the allreduce workspace for quant patterns, initializing if needed.

    Backend is controlled by VLLM_FLASHINFER_ALLREDUCE_BACKEND env var, matching
    non-quant fusion. With ``auto`` this prefers mnnvl and falls back to trtllm
    only on single-node topologies where mnnvl multicast is unavailable.
    """
    global _fi_ar_quant_workspace
    if _fi_ar_quant_workspace is not None:
        return _fi_ar_quant_workspace

    backend, allow_trtllm_fallback = _resolve_fi_ar_backend()

    if get_node_count() > 1 and backend == "trtllm":
        raise ValueError(
            "Flashinfer allreduce quantization fusion is not supported for "
            "multi-node allreduce with 'trtllm' backend. Please use 'mnnvl' "
            "backend instead."
        )

    # Reuse the non-quant workspace if it was already created with the same
    # backend.
    if _fi_ar_workspace is not None and _fi_ar_workspace.backend == backend:
        _fi_ar_quant_workspace = _fi_ar_workspace
        return _fi_ar_quant_workspace

    if (
        _fi_ar_workspace is not None
        and _fi_ar_workspace.backend == "trtllm"
        and allow_trtllm_fallback
        and backend != "trtllm"
    ):
        _fi_ar_quant_workspace = _fi_ar_workspace

View on GitHub (pinned to c794754062)

Solutions

1. Set VLLM_FLASHINFER_ALLREDUCE_BACKEND=mnnvl on all nodes
2. Disable the quant-fusion allreduce optimization and use the standard quantize-then-allreduce path
3. Verify the fabric actually supports multicast (mnnvl) with nvidia-smi nvlink / fabric reports before choosing it

Example fix

# before
export VLLM_FLASHINFER_ALLREDUCE_BACKEND=trtllm  # breaks when TP spans nodes

# after
export VLLM_FLASHINFER_ALLREDUCE_BACKEND=mnnvl

import os
from vllm.distributed import get_node_count
if get_node_count() > 1:
    assert os.environ.get("VLLM_FLASHINFER_ALLREDUCE_BACKEND", "auto") != "trtllm", (
        "quant fusion allreduce needs mnnvl on multi-node")

When it happens

Trigger: Multi-node inference with allreduce+quant fusion enabled (e.g. a TRT-LLM-style quant fusion graph pass) while VLLM_FLASHINFER_ALLREDUCE_BACKEND resolves to 'trtllm', triggering the workspace creation for the quant path.

Common situations: Migrating a working single-node fp8/int4 quant-fusion setup to multi-node without changing the backend env var; 'auto' falling back to trtllm because mnnvl multicast is unavailable on the fabric, then hitting it on a multi-node job.

Related errors

• Flashinfer allreduce is not supported for multi-node allredu
• NCCL error: {error_str}
• VLLM_ROCM_QUICK_REDUCE_QUANTIZATION_MIN_SIZE_KB must be non-
• chat request must contain at least one message
• Recipe deploy_type={deploy_type!r} is not a single-node depl

AI-assisted analysis of vllm-project/vllm@c794754062 (2026-08-14). Data as JSON: /api/errors/2e3fedf4ee50f84a. Report an issue: GitHub.