{"record":{"id":"945e53733269f43c","repo":"huggingface/transformers","slug":"decompose-prefill-decode-expected-at-least-num-ne","errorCode":null,"errorMessage":"decompose_prefill_decode expected at least {num_new_tokens} calls to {type(model).__name__}.forward() during generate(max_new_tokens={num_new_tokens}), but captured {len(calls)}. This likely means generate() bypasses the top-level forward() (e.g. delegates to an inner model), so prefill/decode decomposition is not supported for this architecture.","messagePattern":"decompose_prefill_decode expected at least (.+?) calls to (.+?)\\.forward\\(\\) during generate\\(max_new_tokens=(.+?)\\), but captured (.+?)\\. This likely means generate\\(\\) bypasses the top-level forward\\(\\) \\(e\\.g\\. delegates to an inner model\\), so prefill/decode decomposition is not supported for this architecture\\.","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"src/transformers/exporters/utils.py","lineNumber":774,"sourceCode":"    # `generation_config` alongside generation kwargs is deprecated. Base it on the model's own config\n    # when none is given (preserving its defaults), and deep-copy into a distinct `capture_config` so\n    # the caller's `generation_config` is never mutated.\n    num_new_tokens = 3 if multi_token_decode else 2\n    capture_config = copy.deepcopy(generation_config if generation_config is not None else model.generation_config)\n    capture_config.max_new_tokens = num_new_tokens\n    capture_config.min_new_tokens = num_new_tokens\n    try:\n        with _capture_forward(model) as calls:\n            model.generate(**copy.deepcopy(inputs), generation_config=capture_config)\n    except Exception as e:\n        raise RuntimeError(\n            f\"decompose_prefill_decode failed for {type(model).__name__}. \"\n            f\"Inputs passed: {list(inputs.keys())}. \"\n            f\"Make sure the inputs are compatible with model.generate().\"\n        ) from e\n\n    if len(calls) < num_new_tokens:\n        raise RuntimeError(\n            f\"decompose_prefill_decode expected at least {num_new_tokens} calls to \"\n            f\"{type(model).__name__}.forward() during generate(max_new_tokens={num_new_tokens}), but \"\n            f\"captured {len(calls)}. This likely means generate() bypasses the top-level forward() \"\n            \"(e.g. delegates to an inner model), so prefill/decode decomposition is not supported \"\n            \"for this architecture.\"\n        )\n\n    # Remove `logits_to_keep` from the captured calls — it's a generation-time hint for the model's\n    # internal top-k pruning, not a forward input. The export graph should not depend on it.\n    for call in calls:\n        call.pop(\"logits_to_keep\", None)\n\n    # A single-token decode specializes its query-sequence axis to 1 (never dynamic). When\n    # `multi_token_decode`, merge the two decode steps into one multi-token decode so that axis stays\n    # symbolic (continuation-from-past, or a plain prefill when the cache is empty, and it still covers seq == 1).\n    prefill_inputs = calls[0]\n    decode_inputs = _merge_decode_calls(calls[1:num_new_tokens]) if multi_token_decode else calls[1]\n    return {","sourceCodeStart":756,"sourceCodeEnd":792,"githubUrl":"https://github.com/huggingface/transformers/blob/a597f974857b3d92939971296bc0deb93d33d780/src/transformers/exporters/utils.py#L756-L792","documentation":"decompose_prefill_decode instruments the model's top-level forward() and runs generate(max_new_tokens=2 or 3). It expects at least one call per generated token; if fewer forward() calls were captured, generate() is bypassing the top-level forward (delegating to an inner model), so prefill/decode splitting at this level is impossible and the architecture is unsupported.","triggerScenarios":"Exporting an architecture whose generate() path calls an inner module (e.g. a wrapper model that calls self.model(...)/self.language_model(...)) instead of its own forward; models with custom generate loops or multi-token decode paths that skip the top-level forward.","commonSituations":"Newly added or custom wrapper architectures in an export pipeline; models refactored so forward() is a thin shim; version changes where generation dispatch moved to inner components.","solutions":["Export the inner model directly (e.g. model.model or model.language_model) with decompose_prefill_decode instead of the wrapper","Check for a transformers update where this architecture's export is supported","Fall back to exporting a single non-decomposed generation graph if acceptable"],"exampleFix":"// before\ndecompose_for_generation(wrapper_model, inputs)\n\n// after\n# export the inner causal LM that generate() actually calls\ndecompose_for_generation(wrapper_model.language_model, inner_inputs)","handlingStrategy":"fallback","validationCode":"from transformers.exporters.utils import _capture_forward\n\ndef forwards_are_top_level(model, inputs, n=2) -> bool:\n    with _capture_forward(model) as calls:\n        model(**inputs)\n    return len(calls) >= 1","typeGuard":null,"tryCatchPattern":"try:\n    parts = decompose_for_generation(model, inputs)\nexcept RuntimeError as e:\n    if \"bypasses the top-level forward\" in str(e):\n        parts = decompose_for_generation(model.model, inner_inputs)  # inner module\n    else:\n        raise","preventionTips":["Check whether generate() calls the wrapper's or an inner forward before exporting","Prefer exporting the inner causal LM for wrapper architectures","Track transformers release notes for newly supported export architectures"],"tags":["export","generation","architecture","transformers"],"backgroundTag":null,"analyzedSha":"a597f974857b3d92939971296bc0deb93d33d780","analyzedAt":"2026-08-14T18:24:08.354Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}