{"record":{"id":"a40c9f140a9b3fd6","repo":"huggingface/transformers","slug":"decompose-prefill-decode-failed-for-type-model","errorCode":null,"errorMessage":"decompose_prefill_decode failed for {type(model).__name__}. Inputs passed: {list(inputs.keys())}. Make sure the inputs are compatible with model.generate().","messagePattern":"decompose_prefill_decode failed for (.+?)\\. Inputs passed: (.+?)\\. Make sure the inputs are compatible with model\\.generate\\(\\)\\.","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"src/transformers/exporters/utils.py","lineNumber":767,"sourceCode":"\n    Returns:\n        `dict[str, tuple[torch.nn.Module, dict]]`:\n        `{\"prefill\": (model, prefill_inputs), \"decode\": (model, decode_inputs)}`.\n    \"\"\"\n    # 1 prefill forward + 1 decode (or 2 decode steps merged, when `multi_token_decode`) forward to capture.\n    # Set the capture window on the config itself, not as generate() kwargs — passing a\n    # `generation_config` alongside generation kwargs is deprecated. Base it on the model's own config\n    # when none is given (preserving its defaults), and deep-copy into a distinct `capture_config` so\n    # the caller's `generation_config` is never mutated.\n    num_new_tokens = 3 if multi_token_decode else 2\n    capture_config = copy.deepcopy(generation_config if generation_config is not None else model.generation_config)\n    capture_config.max_new_tokens = num_new_tokens\n    capture_config.min_new_tokens = num_new_tokens\n    try:\n        with _capture_forward(model) as calls:\n            model.generate(**copy.deepcopy(inputs), generation_config=capture_config)\n    except Exception as e:\n        raise RuntimeError(\n            f\"decompose_prefill_decode failed for {type(model).__name__}. \"\n            f\"Inputs passed: {list(inputs.keys())}. \"\n            f\"Make sure the inputs are compatible with model.generate().\"\n        ) from e\n\n    if len(calls) < num_new_tokens:\n        raise RuntimeError(\n            f\"decompose_prefill_decode expected at least {num_new_tokens} calls to \"\n            f\"{type(model).__name__}.forward() during generate(max_new_tokens={num_new_tokens}), but \"\n            f\"captured {len(calls)}. This likely means generate() bypasses the top-level forward() \"\n            \"(e.g. delegates to an inner model), so prefill/decode decomposition is not supported \"\n            \"for this architecture.\"\n        )\n\n    # Remove `logits_to_keep` from the captured calls — it's a generation-time hint for the model's\n    # internal top-k pruning, not a forward input. The export graph should not depend on it.\n    for call in calls:\n        call.pop(\"logits_to_keep\", None)","sourceCodeStart":749,"sourceCodeEnd":785,"githubUrl":"https://github.com/huggingface/transformers/blob/a597f974857b3d92939971296bc0deb93d33d780/src/transformers/exporters/utils.py#L749-L785","documentation":"decompose_prefill_decode runs model.generate() under a forward-call capture to split generation into prefill and decode graphs. If generate() itself raises (bad tokenization, missing generation config fields, wrong input keys, device/dtype errors), the utility wraps the failure in this RuntimeError with the model class and the input keys that were passed. The original exception is chained via 'from e'.","triggerScenarios":"Calling the export decomposition helper with an inputs dict incompatible with model.generate(): wrong key names, unprocessed raw features (e.g. float audio not yet a tensor), inputs on the wrong device, or a generation_config missing required fields.","commonSituations":"Exporting a decoder-only or multimodal model for deployment; passing preprocessor outputs instead of model-ready tensors; generate() failing due to eos/pad token misconfiguration that only surfaces at generation time.","solutions":["Reproduce outside export: call model.generate(**inputs) directly and read the chained __cause__ exception for the real error","Fix the inputs dict so generate() accepts it (correct keys, tensors, device, dtype)","Check model.generation_config (eos_token_id, pad_token_id) is valid for generation","Only then re-run the export decomposition"],"exampleFix":"// before\nexport_packages = decompose_for_generation(model, {\"raw_audio\": wav})\n\n// after\ninputs = processor(wav, sampling_rate=16000, return_tensors=\"pt\")\nexport_packages = decompose_for_generation(model, inputs)","handlingStrategy":"try-catch","validationCode":"# Dry-run generate with the same inputs before export\ndef can_generate(model, inputs) -> bool:\n    try:\n        model.generate(**{k: v.clone() if hasattr(v, \"clone\") else v for k, v in inputs.items()}, max_new_tokens=1)\n        return True\n    except Exception:\n        return False","typeGuard":null,"tryCatchPattern":"try:\n    decompose_for_generation(model, inputs)\nexcept RuntimeError as e:\n    cause = e.__cause__\n    raise RuntimeError(f\"export decomposition failed; root cause: {cause!r}\") from cause","preventionTips":["Always smoke-test model.generate(**inputs) before export","Use the processor to build export inputs, not raw data","Inspect the chained __cause__ rather than the wrapper message"],"tags":["export","generation","inputs","transformers"],"backgroundTag":null,"analyzedSha":"a597f974857b3d92939971296bc0deb93d33d780","analyzedAt":"2026-08-14T18:24:08.354Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}