sgl-project/sglang · error · ValueError

{name} must stay fp32 with curve AdaLN, got {param.dtype}.

Error message

{name} must stay fp32 with curve AdaLN, got {param.dtype}.

What it means

When curve AdaLN is active (adaln_t_table present), every parameter under a '.adaln_proj.linear.' submodule must stay fp32 because the curve interpolation math is dtype-sensitive. post_load_weights scans named_parameters and raises on any such parameter that was downcast during loading.

Source

Thrown at python/sglang/multimodal_gen/runtime/models/dits/minimax_h3.py:2079

        if self.adaln_t_table is not None:
            fp32_param_names = [
                name
                for name in fp32_param_names
                if not name.startswith("time_embedder.")
            ]
            fp32_param_names.append("adaln_t_table")
            if self.adaln_basis is not None:
                fp32_param_names.extend(("adaln_basis", "adaln_mean"))
        for name in fp32_param_names:
            param = self.get_parameter(name)
            if param.dtype != _FP32_DTYPE:
                raise ValueError(
                    f"{name} must stay fp32 after load, got {param.dtype}."
                )
        if self.adaln_t_table is not None:
            for name, param in self.named_parameters():
                if ".adaln_proj.linear." in name and param.dtype != _FP32_DTYPE:
                    raise ValueError(
                        f"{name} must stay fp32 with curve AdaLN, got {param.dtype}."
                    )
        # assign=True loading may re-register this persistent buffer as a parameter
        rope_inv_freq = self.rope.inv_freq
        if rope_inv_freq.dtype != _FP32_DTYPE:
            raise ValueError(
                f"rope.inv_freq must stay fp32 after load, got {rope_inv_freq.dtype}."
            )
        if self.adaln_cache is not None:
            self.adaln_cache.load(self.video_patch_proj.weight.device)

    def _time_embedding(self, timesteps: torch.Tensor) -> torch.Tensor:
        if self.adaln_t_table is None:
            assert self.time_embedder is not None
            return self.time_embedder(timesteps)

        grid = self.adaln_t_table.shape[0]
        position = timesteps.to(_FP32_DTYPE).clamp(0, 1) * (grid - 1)

View on GitHub (pinned to 0132848349)

Solutions

  1. Exclude adaln_proj.linear.* tensors from any dtype casting in your conversion/loading pipeline
  2. Re-save the checkpoint keeping adaln_proj weights in fp32
  3. Load in the native checkpoint dtype and cast only compute weights afterwards

Example fix

# before
model.to(torch.bfloat16); model.load_state_dict(sd, assign=True)
# after
model.load_state_dict(sd, assign=True)
model._keep_adaln_fp32()  # or cast everything except adaln_proj/adaln tables
Defensive patterns

Strategy: type-guard

Validate before calling

for name, p in model.named_parameters():
    if ".adaln_proj.linear." in name:
        assert p.dtype == torch.float32, (name, p.dtype)

Type guard

def adaln_proj_fp32(model) -> bool:
    return all(p.dtype == torch.float32
               for n, p in model.named_parameters()
               if ".adaln_proj.linear." in n)

Try / catch

try:
    model.post_load_weights()
except ValueError:
    with torch.no_grad():
        for n, p in model.named_parameters():
            if ".adaln_proj.linear." in n:
                p.data = p.data.float()
    model.post_load_weights()

Prevention

When it happens

Trigger: Loading a curve-AdaLN checkpoint where adaln_proj.linear.weight/bias arrived as bf16/fp16 — usually from a global dtype cast in the loader or a mixed-precision checkpoint.

Common situations: model.to(torch.bfloat16) before load; converters that cast all linear weights; fp16-serialized checkpoints.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/8d2c3b482d0db10a. Report an issue: GitHub.