{"record":{"id":"f8dbd58d6693442b","repo":"sgl-project/sglang","slug":"rope-inv-freq-must-stay-fp32-after-load-got-rope","errorCode":null,"errorMessage":"rope.inv_freq must stay fp32 after load, got {rope_inv_freq.dtype}.","messagePattern":"rope\\.inv_freq must stay fp32 after load, got (.+?)\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/multimodal_gen/runtime/models/dits/minimax_h3.py","lineNumber":2085,"sourceCode":"            fp32_param_names.append(\"adaln_t_table\")\n            if self.adaln_basis is not None:\n                fp32_param_names.extend((\"adaln_basis\", \"adaln_mean\"))\n        for name in fp32_param_names:\n            param = self.get_parameter(name)\n            if param.dtype != _FP32_DTYPE:\n                raise ValueError(\n                    f\"{name} must stay fp32 after load, got {param.dtype}.\"\n                )\n        if self.adaln_t_table is not None:\n            for name, param in self.named_parameters():\n                if \".adaln_proj.linear.\" in name and param.dtype != _FP32_DTYPE:\n                    raise ValueError(\n                        f\"{name} must stay fp32 with curve AdaLN, got {param.dtype}.\"\n                    )\n        # assign=True loading may re-register this persistent buffer as a parameter\n        rope_inv_freq = self.rope.inv_freq\n        if rope_inv_freq.dtype != _FP32_DTYPE:\n            raise ValueError(\n                f\"rope.inv_freq must stay fp32 after load, got {rope_inv_freq.dtype}.\"\n            )\n        if self.adaln_cache is not None:\n            self.adaln_cache.load(self.video_patch_proj.weight.device)\n\n    def _time_embedding(self, timesteps: torch.Tensor) -> torch.Tensor:\n        if self.adaln_t_table is None:\n            assert self.time_embedder is not None\n            return self.time_embedder(timesteps)\n\n        grid = self.adaln_t_table.shape[0]\n        position = timesteps.to(_FP32_DTYPE).clamp(0, 1) * (grid - 1)\n        lower = position.floor().clamp(max=grid - 2).to(torch.long)\n        fraction = (position - lower).unsqueeze(-1)\n        lower_value = self.adaln_t_table.index_select(0, lower)\n        upper_value = self.adaln_t_table.index_select(0, lower + 1)\n        return torch.lerp(lower_value, upper_value, fraction)\n","sourceCodeStart":2067,"sourceCodeEnd":2103,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/multimodal_gen/runtime/models/dits/minimax_h3.py#L2067-L2103","documentation":"The rotary-embedding inverse-frequency buffer (rope.inv_freq) must remain fp32; with assign=True loading PyTorch may re-register this persistent buffer from the incoming state_dict, and if that tensor is bf16/fp16 the post-load check fails. fp32 inv_freq is required for positional-frequency numerical accuracy.","triggerScenarios":"Loading a state_dict with assign=True where 'rope.inv_freq' was saved in a reduced dtype, or where load re-registers it as a parameter with the wrong dtype.","commonSituations":"Half-precision checkpoint exports that included inv_freq; custom loaders that materialize all tensors in the compute dtype; newer PyTorch behavior changes around persistent buffers with assign=True.","solutions":["Re-save the checkpoint with rope.inv_freq in fp32, or drop inv_freq from the state_dict so the module's own fp32 buffer survives","After assign=True loading, explicitly restore: model.rope.inv_freq = model.rope.inv_freq.float() before calling post_load_weights","Avoid global dtype casts when materializing state_dict tensors"],"exampleFix":"# before\nsd = torch.load(p, map_location=\"cpu\", dtype=torch.bfloat16)  # casts inv_freq\n# after\nsd = torch.load(p, map_location=\"cpu\")\nsd.pop(\"rope.inv_freq\", None)  # let module keep its fp32 buffer","handlingStrategy":"type-guard","validationCode":"sd.pop(\"rope.inv_freq\", None)\nassert model.rope.inv_freq.dtype == torch.float32","typeGuard":"def inv_freq_fp32(model) -> bool:\n    return model.rope.inv_freq.dtype == torch.float32","tryCatchPattern":"try:\n    model.post_load_weights()\nexcept ValueError:\n    model.rope.inv_freq = model.rope.inv_freq.float()\n    model.post_load_weights()","preventionTips":["Exclude rope.inv_freq from dtype casts and checkpoints","After assign=True loads, re-assert buffer dtypes"],"tags":["dtype","fp32","rope","weight-loading"],"backgroundTag":"weight-dtype-mismatch","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}