sgl-project/sglang · error · ValueError

Deterministic inference with absorbed-MLA models on the fa4

Error message

Deterministic inference with absorbed-MLA models on the fa4 attention backend requires SM100/SM110: it runs absorbed MLA, whose qv argument flash_attn.cute only implements on those archs.

What it means

Raised when deterministic inference with an absorbed-MLA model selects the fa4 (flash_attn.cute) backend on hardware that is not SM100/SM110 (Blackwell). The absorbed-MLA qv argument in flash_attn.cute is only implemented for those architectures, so older GPUs cannot run this path deterministically.

Source

Thrown at python/sglang/srt/server_args.py:9238

                        "Glm4MoeLiteForCausalLM",
                    ]
                except Exception:
                    pass

            # Check attention backend
            run_post_process_pass(self, _deterministic_attention_backend)

            attention_backend = resolved_view(self).attention_backend
            if is_deepseek_model:
                if (
                    attention_backend
                    not in RADIX_SUPPORTED_DETERMINISTIC_ATTENTION_BACKEND
                ):
                    raise ValueError(
                        f"Currently only {RADIX_SUPPORTED_DETERMINISTIC_ATTENTION_BACKEND} attention backends are supported for deterministic inference with absorbed-MLA models. But you're using {attention_backend}."
                    )
                if attention_backend == "fa4" and not is_sm100_or_sm110_supported():
                    raise ValueError(
                        "Deterministic inference with absorbed-MLA models on the fa4 "
                        "attention backend requires SM100/SM110: it runs "
                        "absorbed MLA, whose qv argument flash_attn.cute only "
                        "implements on those archs."
                    )

            if attention_backend not in RADIX_SUPPORTED_DETERMINISTIC_ATTENTION_BACKEND:
                # Currently, only certain backends support radix cache. Support for other backends is in progress
                self._declare(
                    "_handle_deterministic_inference",
                    disable_radix_cache=True,
                )
                logger.warning(
                    f"Currently radix cache is not compatible with {attention_backend} attention backend for deterministic inference. It will be supported in the future."
                )

            # Check TP size
            if cfg.tp_size > 1:

View on GitHub (pinned to 0132848349)

Solutions

  1. Use a different supported attention backend (e.g. fa3) on non-Blackwell hardware
  2. Or run on SM100/SM110 (B200/GB200-class) GPUs
  3. Or disable deterministic inference if not strictly needed

Example fix

# before (on H100)
--attention-backend fa4 --rl-on-policy-target ...
# after
--attention-backend fa3 --rl-on-policy-target ...
Defensive patterns

Strategy: fallback

Validate before calling

import torch
sm = torch.cuda.get_device_capability()
if args.attention_backend == 'fa4' and deterministic and sm not in ((10,0),(11,0)):
    args.attention_backend = 'fa3'

Type guard

def fa4_ok_for_this_gpu() -> bool:
    import torch
    cc = torch.cuda.get_device_capability()
    return cc[0] == 10 and cc[1] in (0, 1) or cc == (11, 0)

Try / catch

except ValueError as e:
    if 'SM100/SM110' in str(e):
        retry with --attention-backend fa3

Prevention

When it happens

Trigger: Running with --attention-backend fa4 plus deterministic inference on a DeepSeek model on Hopper (H100) or older GPUs; the preceding backend check passes (fa4 is in the supported set) but the arch check fails.

Common situations: Developing on H100 with configs written for B200/Blackwell; CI runners with different GPU archs than the target cluster; fa4 newly supported for determinism but only on SM100+.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/da18d3a83a2bf6ee. Report an issue: GitHub.