{"record":{"id":"35ac827bf35aa8ec","repo":"huggingface/transformers","slug":"aten-grouped-mm-number-of-experts-mat-b-shape-0","errorCode":null,"errorMessage":"_aten_grouped_mm: number of experts (mat_b.shape[0]) must be static at translation time","messagePattern":"_aten_grouped_mm: number of experts \\(mat_b\\.shape\\[0\\]\\) must be static at translation time","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/transformers/exporters/exporter_onnx.py","lineNumber":847,"sourceCode":"    one_hot = op.OneHot(self, depth, op.Constant(value_ints=[0, 1]), axis=-1)\n    return op.ReduceSum(one_hot, op.Constant(value_ints=[0]), keepdims=0)\n\n\ndef _aten_grouped_mm(mat_a: TReal, mat_b: TReal, offs: INT64, bias=None, out_dtype=None) -> TReal:\n    \"\"\"ONNX implementation of `aten._grouped_mm.default`.\n\n    `_grouped_mm(mat_a: (M, K), mat_b: (G, K, N), offs: (G,))` computes `out[r] =\n    mat_a[r] @ mat_b[group(r)]` where rows are sorted by group and `offs` holds the\n    cumulative end index per group.\n\n    Per-group `Slice + MatMul + Concat`. `G` (number of experts) is static for any\n    concrete model, so unroll at translation time: emit one `Slice + MatMul` triple\n    per group and a final `Concat`. Avoids the `(M, K, N)` materialisation a naive\n    `weight[group_idx]` gather would emit — peak memory is `O(M·N + max(n_g)·K + K·N)`.\n    \"\"\"\n    G = mat_b.shape[0]\n    if not isinstance(G, int):\n        raise ValueError(\"_aten_grouped_mm: number of experts (mat_b.shape[0]) must be static at translation time\")\n\n    offs_i64 = op.Cast(offs, to=7)\n    axes_0 = op.Constant(value_ints=[0])\n    zero_1d = op.Constant(value_ints=[0])\n\n    outputs = []\n    prev_end = zero_1d\n    for g in range(G):\n        g_lo = op.Constant(value_ints=[g])\n        g_hi = op.Constant(value_ints=[g + 1])\n        end = op.Slice(offs_i64, g_lo, g_hi, axes_0)  # (1,) — offs[g]\n        a_g = op.Slice(mat_a, prev_end, end, axes_0)  # (n_g, K)\n        w_g = op.Squeeze(op.Slice(mat_b, g_lo, g_hi, axes_0), axes_0)  # (K, N)\n        outputs.append(op.MatMul(a_g, w_g))  # (n_g, N)\n        prev_end = end\n\n    return op.Concat(*outputs, axis=0)  # (M, N)\n","sourceCodeStart":829,"sourceCodeEnd":865,"githubUrl":"https://github.com/huggingface/transformers/blob/a597f974857b3d92939971296bc0deb93d33d780/src/transformers/exporters/exporter_onnx.py#L829-L865","documentation":"The ONNX translation of torch._grouped_mm (used by MoE experts) unrolls the grouped matmul into per-group Slice+MatMul at translation time. It needs the number of experts G = mat_b.shape[0] as a concrete int; if the axis is symbolic (marked dynamic, or an unbacked SymInt from data-dependent slicing) it cannot unroll and raises this ValueError.","triggerScenarios":"Exporting a Mixture-of-Experts model to ONNX while dynamic_shapes marks the experts dimension dynamic (e.g. blanket Dim.AUTO from dynamic=True), leaving mat_b.shape[0] a SymInt instead of an int.","commonSituations":"Using DynamoConfig(dynamic=True) with no explicit dynamic_shapes — the exporter even warns that this marks every axis symbolic; exporting splinter/MoE models where G accidentally falls into the dynamic set.","solutions":["Pass explicit dynamic_shapes that leave the experts (mat_b.shape[0]) axis static; only mark truly varying axes (batch, sequence) dynamic.","If using dynamic=True alone, switch to dynamic=True plus dynamic_shapes with Dim.AUTO only on batch/seq.","For ONNX specifically, prefer fully static shapes for MoE weights — expert count is fixed per checkpoint."],"exampleFix":"# before\ncfg = DynamoConfig(dynamic=True)  # experts dim becomes symbolic -> ValueError at ONNX translation\n\n# after\nfrom torch.export import Dim\nseq = Dim(\"seq\", min=1, max=4096)\ncfg = DynamoConfig(\n    dynamic=True,\n    dynamic_shapes={\"hidden_states\": {0: Dim.AUTO, 1: seq}},  # experts dim stays static\n)","handlingStrategy":"validation","validationCode":"# Before ONNX export of a MoE model, ensure the experts dim is static.\nfrom torch.export import Dim\n\ncfg = DynamoConfig(\n    dynamic=True,\n    dynamic_shapes={\n        \"hidden_states\": {0: Dim.AUTO, 1: Dim(\"seq\")},  # expert weights left static\n        # do NOT mark any dim of router_logits/expert weights dynamic\n    },\n)","typeGuard":"def experts_dim_is_static(mat_b: torch.Tensor) -> bool:\n    return isinstance(mat_b.shape[0], int)","tryCatchPattern":"try:\n    OnnxExporter().export(model, inputs, cfg)\nexcept ValueError as e:\n    if \"number of experts\" in str(e) and \"static\" in str(e):\n        cfg = DynamoConfig(dynamic=False)  # fully static retry — expert count is fixed per checkpoint\n        OnnxExporter().export(model, inputs, cfg)\n    else:\n        raise","preventionTips":["Never use bare dynamic=True for MoE/ONNX exports; always supply targeted dynamic_shapes","Keep expert-count and num_heads/head_dim axes static in every export config","Add a canary MoE model to CI to catch symbolic-expert regressions early"],"tags":["export","onnx","moe","dynamic-shapes"],"backgroundTag":null,"analyzedSha":"a597f974857b3d92939971296bc0deb93d33d780","analyzedAt":"2026-08-14T18:24:08.354Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}