{"record":{"id":"63ab3eb6fe3511de","repo":"sgl-project/sglang","slug":"name-must-stay-fp32-after-load-got-param-dtype","errorCode":null,"errorMessage":"{name} must stay fp32 after load, got {param.dtype}.","messagePattern":"(.+?) must stay fp32 after load, got (.+?)\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/multimodal_gen/runtime/models/dits/minimax_h3.py","lineNumber":2073,"sourceCode":"            # \"ones\"); claiming only undeclared params keeps that intact.\n            if getattr(param, \"missing_param_init\", None) is None:\n                param.missing_param_init = \"error\"\n\n    def post_load_weights(self) -> None:\n        fp32_param_names = list(_MINIMAX_H3_FP32_PARAM_NAMES_IN_MODEL_ORDER)\n        if self.adaln_t_table is not None:\n            fp32_param_names = [\n                name\n                for name in fp32_param_names\n                if not name.startswith(\"time_embedder.\")\n            ]\n            fp32_param_names.append(\"adaln_t_table\")\n            if self.adaln_basis is not None:\n                fp32_param_names.extend((\"adaln_basis\", \"adaln_mean\"))\n        for name in fp32_param_names:\n            param = self.get_parameter(name)\n            if param.dtype != _FP32_DTYPE:\n                raise ValueError(\n                    f\"{name} must stay fp32 after load, got {param.dtype}.\"\n                )\n        if self.adaln_t_table is not None:\n            for name, param in self.named_parameters():\n                if \".adaln_proj.linear.\" in name and param.dtype != _FP32_DTYPE:\n                    raise ValueError(\n                        f\"{name} must stay fp32 with curve AdaLN, got {param.dtype}.\"\n                    )\n        # assign=True loading may re-register this persistent buffer as a parameter\n        rope_inv_freq = self.rope.inv_freq\n        if rope_inv_freq.dtype != _FP32_DTYPE:\n            raise ValueError(\n                f\"rope.inv_freq must stay fp32 after load, got {rope_inv_freq.dtype}.\"\n            )\n        if self.adaln_cache is not None:\n            self.adaln_cache.load(self.video_patch_proj.weight.device)\n\n    def _time_embedding(self, timesteps: torch.Tensor) -> torch.Tensor:","sourceCodeStart":2055,"sourceCodeEnd":2091,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/multimodal_gen/runtime/models/dits/minimax_h3.py#L2055-L2091","documentation":"post_load_weights verifies that numerically sensitive parameters (the AdaLN t-table, adaln_basis, adaln_mean, and similar fp32-mandatory params) remain torch.float32 after weight loading. If weight loading (e.g. a state_dict cast to bf16, or assign=True with a dtype-converted tensor) downcast them, the error names the offending parameter and its dtype.","triggerScenarios":"Loading a checkpoint whose fp32 params (adaln_t_table etc.) were saved/loaded as bf16/fp16; using a loader that casts the whole state_dict to the compute dtype before assign.","commonSituations":"Pre-casting checkpoints to bf16 to shrink files; custom load hooks applying model.to(torch.bfloat16) before post_load_weights; conversion scripts that rewrite tensors with torch.Tensor.to(dtype).","solutions":["Keep the fp32-mandatory tensors fp32 in the checkpoint (re-save without global dtype casting)","Load with assign but cast only non-fp32-mandatory params, letting post_load_weights re-assert dtypes","If converting, whitelist: adaln_t_table, adaln_basis, adaln_mean (and other listed names) from any dtype downcast"],"exampleFix":"# before\nstate = {k: v.to(torch.bfloat16) for k, v in state.items()}\nmodel.load_state_dict(state, assign=True)\n# after\nFP32_KEEP = {\"adaln_t_table\", \"adaln_basis\", \"adaln_mean\"}\nstate = {k: (v if k in FP32_KEEP else v.to(torch.bfloat16)) for k, v in state.items()}","handlingStrategy":"type-guard","validationCode":"FP32_KEEP = {\"adaln_t_table\", \"adaln_basis\", \"adaln_mean\"}\nfor name, p in model.named_parameters():\n    if name in FP32_KEEP:\n        assert p.dtype == torch.float32, (name, p.dtype)","typeGuard":"def fp32_params_ok(model) -> bool:\n    keep = {\"adaln_t_table\", \"adaln_basis\", \"adaln_mean\"}\n    return all(p.dtype == torch.float32 for n, p in model.named_parameters() if n in keep)","tryCatchPattern":"try:\n    model.post_load_weights()\nexcept ValueError as e:\n    for n, p in model.named_parameters():\n        if n in FP32_KEEP:\n            p.data = p.data.float()\n    model.post_load_weights()","preventionTips":["Never globally cast state_dicts to bf16","Whitelist fp32-mandatory tensors in conversion scripts"],"tags":["dtype","fp32","weight-loading","adaln"],"backgroundTag":"weight-dtype-mismatch","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}