sgl-project/sglang · error · ValueError

Kimi-K3 GGUF ssm_a must contain finite floating values

Error message

Kimi-K3 GGUF ssm_a must contain finite floating values

What it means

Raised by _kda_a_log_target_value when undoing llama.cpp's GGUF-time A_log -> -exp(A_log) transform on a tensor that is not floating point or contains non-finite values (NaN/Inf). The KDA/SSM A_log recovery via torch.log(-raw) requires finite floats, so malformed or integer-quantized ssm_a data is rejected.

Source

Thrown at python/sglang/srt/model_loader/kimi_k3_gguf.py:132

    return checkpoint_name.removesuffix("weight") + "qweight"


def _residual_target_value(raw: torch.Tensor, target_index: int) -> torch.Tensor:
    if raw.ndim != 1:
        raise ValueError(
            f"Kimi-K3 attention-residual score must be a vector, got {tuple(raw.shape)}"
        )
    if target_index == 0:
        return raw.unsqueeze(0)
    if target_index == 1:
        return torch.ones_like(raw)
    raise ValueError(f"invalid Kimi-K3 attention-residual target {target_index}")


def _kda_a_log_target_value(raw: torch.Tensor) -> torch.Tensor:
    """Undo llama.cpp's GGUF-time ``A_log -> -exp(A_log)`` transform."""
    if not raw.is_floating_point() or not torch.isfinite(raw).all():
        raise ValueError("Kimi-K3 GGUF ssm_a must contain finite floating values")
    if not torch.all(raw < 0):
        raise ValueError("Kimi-K3 GGUF ssm_a must contain only -exp(A_log) values")
    return torch.log(-raw)


def kimi_k3_nonexpert_weights_iterator(
    manifest_path: str | os.PathLike[str],
) -> Generator[tuple[str, torch.Tensor], None, None]:
    """Stream non-routed tensors shard by shard without reading routed payloads."""

    import gguf

    manifest_file = Path(manifest_path).resolve()
    manifest = json.loads(manifest_file.read_text(encoding="utf-8"))
    if manifest.get("format") != "SGLANG-KIMI-GGMLMOEPACK-ADAPTER-v1":
        raise ValueError("Kimi-K3 manifest format is unsupported")
    if not manifest.get("complete"):
        raise ValueError("Kimi-K3 manifest is incomplete")

View on GitHub (pinned to 0132848349)

Solutions

Re-export or dequantize the GGUF so ssm_a (A_log) is stored as F32/F16/BF16 with finite values.
Verify shard integrity with gguf-py or the manifest size check before loading.
If NaNs persist, the source GGUF is corrupt — regenerate it from the original checkpoint.

Example fix

# before: ssm_a stored quantized -> raises
# after: keep ssm_a unquantized when writing GGUF
writer.add_tensor("ssm_a", a_log_float32.numpy())  # F32, finite

Defensive patterns

Strategy: validation

Validate before calling

raw = torch.tensor(tensor.data)
assert raw.is_floating_point() and torch.isfinite(raw).all(), "ssm_a not finite floats"

Type guard

def is_finite_float(t: torch.Tensor) -> bool:
    return t.is_floating_point() and bool(torch.isfinite(t).all())

Prevention

Keep ssm_a unquantized (F32/F16/BF16) when writing GGUF.
Validate GGUF tensors for finiteness right after reading the shard.

When it happens

Trigger: kimi_k3_nonexpert_weights_iterator yields an ssm_a tensor that was stored as an integer/quantized GGUF type, or contains NaN/±Inf, and it is routed through _kda_a_log_target_value.

Common situations: Converting a Kimi-K3 GGUF file where ssm_a was quantized (e.g. to Q8) by the GGUF writer; corrupted shard file producing NaNs; a partial/interrupted GGUF conversion.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/4df3f65fc9382e8c. Report an issue: GitHub.