{"record":{"id":"2e3fedf4ee50f84a","repo":"vllm-project/vllm","slug":"flashinfer-allreduce-quantization-fusion-is-not-su","errorCode":null,"errorMessage":"Flashinfer allreduce quantization fusion is not supported for multi-node allreduce with 'trtllm' backend. Please use 'mnnvl' backend instead.","messagePattern":"Flashinfer allreduce quantization fusion is not supported for multi-node allreduce with 'trtllm' backend\\. Please use 'mnnvl' backend instead\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"vllm/distributed/device_communicators/flashinfer_all_reduce.py","lineNumber":221,"sourceCode":"    hidden_dim: int,\n    dtype: torch.dtype,\n    group: ProcessGroup,\n):\n    \"\"\"\n    Return the allreduce workspace for quant patterns, initializing if needed.\n\n    Backend is controlled by VLLM_FLASHINFER_ALLREDUCE_BACKEND env var, matching\n    non-quant fusion. With ``auto`` this prefers mnnvl and falls back to trtllm\n    only on single-node topologies where mnnvl multicast is unavailable.\n    \"\"\"\n    global _fi_ar_quant_workspace\n    if _fi_ar_quant_workspace is not None:\n        return _fi_ar_quant_workspace\n\n    backend, allow_trtllm_fallback = _resolve_fi_ar_backend()\n\n    if get_node_count() > 1 and backend == \"trtllm\":\n        raise ValueError(\n            \"Flashinfer allreduce quantization fusion is not supported for \"\n            \"multi-node allreduce with 'trtllm' backend. Please use 'mnnvl' \"\n            \"backend instead.\"\n        )\n\n    # Reuse the non-quant workspace if it was already created with the same\n    # backend.\n    if _fi_ar_workspace is not None and _fi_ar_workspace.backend == backend:\n        _fi_ar_quant_workspace = _fi_ar_workspace\n        return _fi_ar_quant_workspace\n\n    if (\n        _fi_ar_workspace is not None\n        and _fi_ar_workspace.backend == \"trtllm\"\n        and allow_trtllm_fallback\n        and backend != \"trtllm\"\n    ):\n        _fi_ar_quant_workspace = _fi_ar_workspace","sourceCodeStart":203,"sourceCodeEnd":239,"githubUrl":"https://github.com/vllm-project/vllm/blob/c794754062d49a8fdb63ab3c5215b488b865030c/vllm/distributed/device_communicators/flashinfer_all_reduce.py#L203-L239","documentation":"get_flashinfer_allreduce_quant_workspace is the quantization-fusion variant of the FlashInfer allreduce workspace (allreduce + per-tensor quant fused). It enforces the same topology constraint as the non-quant path: the 'trtllm' backend cannot drive cross-node allreduce, so backend == 'trtllm' with get_node_count() > 1 raises ValueError and directs you to 'mnnvl'.","triggerScenarios":"Multi-node inference with allreduce+quant fusion enabled (e.g. a TRT-LLM-style quant fusion graph pass) while VLLM_FLASHINFER_ALLREDUCE_BACKEND resolves to 'trtllm', triggering the workspace creation for the quant path.","commonSituations":"Migrating a working single-node fp8/int4 quant-fusion setup to multi-node without changing the backend env var; 'auto' falling back to trtllm because mnnvl multicast is unavailable on the fabric, then hitting it on a multi-node job.","solutions":["Set VLLM_FLASHINFER_ALLREDUCE_BACKEND=mnnvl on all nodes","Disable the quant-fusion allreduce optimization and use the standard quantize-then-allreduce path","Verify the fabric actually supports multicast (mnnvl) with nvidia-smi nvlink / fabric reports before choosing it"],"exampleFix":"# before\nexport VLLM_FLASHINFER_ALLREDUCE_BACKEND=trtllm  # breaks when TP spans nodes\n\n# after\nexport VLLM_FLASHINFER_ALLREDUCE_BACKEND=mnnvl","handlingStrategy":"validation","validationCode":"import os\nfrom vllm.distributed import get_node_count\nif get_node_count() > 1:\n    assert os.environ.get(\"VLLM_FLASHINFER_ALLREDUCE_BACKEND\", \"auto\") != \"trtllm\", (\n        \"quant fusion allreduce needs mnnvl on multi-node\")","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Centralize the backend env var in one deployment template","Validate topology (node count) at startup before graph capture","Keep single-node and multi-node configs separate"],"tags":["distributed","flashinfer","allreduce","quantization","multi-node"],"backgroundTag":null,"analyzedSha":"c794754062d49a8fdb63ab3c5215b488b865030c","analyzedAt":"2026-08-14T21:17:39.825Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}