{"record":{"id":"d7fc9435f14b6607","repo":"sgl-project/sglang","slug":"embed-tokens-weight-has-shape-embed-shape-but-co","errorCode":null,"errorMessage":"embed_tokens.weight has shape {embed_shape} but config says (vocab_size, hidden_size) = ({self.args.vocab_size}, {hidden})","messagePattern":"embed_tokens\\.weight has shape (.+?) but config says \\(vocab_size, hidden_size\\) = \\((.+?), (.+?)\\)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/hardware_backend/mlx/models/muse_glimmer_mlx.py","lineNumber":669,"sourceCode":"                    \" (weights look already fused: if this is a packaged \"\n                    'artifact, its config.json must carry \"muse_glimmer_mlx_format\": '\n                    f\"{MUSE_GLIMMER_MLX_FORMAT_VERSION})\"\n                )\n            raise ValueError(\n                \"not a complete raw Muse Glimmer HF checkpoint: \"\n                f\"{len(missing)} missing keys {missing[:4]}\"\n                f\"{'...' if len(missing) > 4 else ''}, \"\n                f\"{len(unexpected)} unexpected keys {unexpected[:4]}\"\n                f\"{'...' if len(unexpected) > 4 else ''}{hint}\"\n            )\n\n        H = self.args.num_attention_heads\n        D = self.args.head_dim\n        hidden = self.args.hidden_size\n\n        embed_shape = tuple(weights[\"model.embed_tokens.weight\"].shape)\n        if embed_shape != (self.args.vocab_size, hidden):\n            raise ValueError(\n                f\"embed_tokens.weight has shape {embed_shape} but config says \"\n                f\"(vocab_size, hidden_size) = ({self.args.vocab_size}, {hidden})\"\n            )\n        raw_q_shape = tuple(weights[\"model.layers.0.self_attn.q_proj.weight\"].shape)\n        if raw_q_shape != (H * D, hidden):\n            raise ValueError(\n                f\"raw q_proj.weight has shape {raw_q_shape}, expected \"\n                f\"({H * D}, {hidden}); a width of {2 * H * D} means the gate \"\n                \"is already fused — such artifacts must carry \"\n                f'\"muse_glimmer_mlx_format\": {MUSE_GLIMMER_MLX_FORMAT_VERSION} in config.json'\n            )\n\n        new_weights = {}\n        for name, w in weights.items():\n            # mlx derives RoPE itself; drop cached buffers.\n            if \"rotary_emb\" in name:\n                continue\n            if any(marker in name for marker in _VISION_KEY_MARKERS):","sourceCodeStart":651,"sourceCodeEnd":687,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/hardware_backend/mlx/models/muse_glimmer_mlx.py#L651-L687","documentation":"During sanitize, the shape of model.embed_tokens.weight is compared against (vocab_size, hidden_size) from the config args. Any mismatch — different vocab, different hidden size, or transposed/damaged tensor — raises this error before any compute happens.","triggerScenarios":"Loading weights whose embedding matrix dimensions don't match config.json, e.g. vocab_size=32000 in config but a 128256-row embedding, or hidden_size edited by hand.","commonSituations":"Mixing config.json from one model revision with weights from another (common with quantized/merged variants), typos in vocab_size/hidden_size, or truncated shard downloads.","solutions":["Fix config.json's vocab_size/hidden_size to match the actual embedding tensor","Re-download the correct, unmodified checkpoint+config pair","If the tensor is transposed/corrupted, re-export the safetensors"],"exampleFix":"// before: config vocab_size=32000, embed shape (128256, 4096)\n// after: \"vocab_size\": 128256 in config.json","handlingStrategy":"validation","validationCode":"emb = weights[\"model.embed_tokens.weight\"].shape\nassert tuple(emb) == (cfg[\"vocab_size\"], cfg[\"hidden_size\"])","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Never mix config.json across model revisions","Checksum-verify downloaded shards"],"tags":["shape-mismatch","config-validation","embedding"],"backgroundTag":"tensor-shape-mismatch","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}