sgl-project/sglang · critical · ImportError

Inkling relative attention requires the vendored FA4 CUTE in

Error message

Inkling relative attention requires the vendored FA4 CUTE interface.

What it means

Raised when the Inkling relative-attention score_mod is requested but the vendored FlashAttention-4 CUTE (CuTe DSL) interface failed to import (cute, Float32, or SeqlenInfoQK is None). The module caches the import error and chains it into this ImportError, so relative attention bias cannot be built. It indicates the FA4/CuTe dependency or its supported platform is missing in this environment.

Source

Thrown at python/sglang/srt/models/inkling_common/attn.py:61

try:
    import cutlass.cute as cute
    from cutlass.cute import Float32

    from sglang.kernels.ops.attention.flash_attn.cute.seqlen_info import SeqlenInfoQK
except Exception as _import_error:
    cute = None
    Float32 = None
    SeqlenInfoQK = None
    _cute_import_error = _import_error
else:
    _cute_import_error = None


@cache
def get_inkling_relative_attention_score_mod(rel_extent: int) -> Callable:
    if cute is None or Float32 is None or SeqlenInfoQK is None:
        raise ImportError(
            "Inkling relative attention requires the vendored FA4 CUTE interface."
        ) from _cute_import_error

    @cute.jit
    def score_mod_rel_bias(
        scores: cute.TensorSSA,
        b_idx: cute.TensorSSA,
        h_idx: cute.TensorSSA,
        q_idx: cute.TensorSSA,
        kv_idx: cute.TensorSSA,
        seqlen_info: SeqlenInfoQK,
        aux_tensors: list[cute.Tensor],
    ) -> cute.TensorSSA:
        rel_logits = aux_tensors[0]

        seqlen_local_offset = seqlen_info.seqlen_k - seqlen_info.seqlen_q
        rel_dist = (q_idx + seqlen_local_offset) - kv_idx
        global_q_idx = seqlen_info.offset_q + q_idx

View on GitHub (pinned to 0132848349)

Solutions

Upgrade sgl-kernel (and sglang) to a version that ships the vendored FA4 CUTE interface and reinstall: pip install -U sglang[all] sgl-kernel
Verify a CUDA GPU is visible (torch.cuda.is_available()) and you are on Linux x86_64; the CUTE path is CUDA-only
Disable the Inkling relative-attention feature / use an attention backend that does not require the CUTE score_mod
If building from source, rebuild sgl-kernel with FA4 enabled and check the original _cute_import_error chained below this message

Example fix

# before: run Inkling with rel attention on CPU / old wheel -> ImportError
# after: ensure CUDA + updated kernel
pip install -U "sglang[all]" sgl-kernel
python -c "import torch; assert torch.cuda.is_available()"

Defensive patterns

Strategy: fallback

Validate before calling

from sglang.srt.models.inkling_common import attn
def cute_available() -> bool:
    return not (attn.cute is None or attn.Float32 is None or attn.SeqlenInfoQK is None)
if cute_available():
    mod = attn.get_inkling_relative_attention_score_mod(extent)

Try / catch

except ImportError as e: log warning; fall back to a non-relative attention backend or abort with a clear 'requires FA4 CUTE / CUDA' message.

Prevention

Pin sglang + sgl-kernel versions that ship the vendored FA4 CUTE interface
Gate relative-attention features on torch.cuda.is_available() in launcher code
Smoke-test the CUTE import at startup before serving traffic

When it happens

Trigger: Calling forward -> get_inkling_relative_attention_score_mod(rel_extent) on a GPU/config that enables Inkling relative attention when the vendored sgl_kernel FA4 CUTE symbols (cute, Float32, SeqlenInfoQK) were not importable (non-CUDA platform, sgl_kernel built without FA4, old wheel).

Common situations: Running an Inkling model with relative attention on CPU or macOS; using a stale sgl-kernel/sglang wheel that predates the vendored FA4 CUTE interface; broken CUDA install; importing after a partial install.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/84fdc7bffde1029a. Report an issue: GitHub.