huggingface/transformers · error · RuntimeError

decompose_prefill_decode failed for {type(model).__name__}.

Error message

decompose_prefill_decode failed for {type(model).__name__}. Inputs passed: {list(inputs.keys())}. Make sure the inputs are compatible with model.generate().

What it means

decompose_prefill_decode runs model.generate() under a forward-call capture to split generation into prefill and decode graphs. If generate() itself raises (bad tokenization, missing generation config fields, wrong input keys, device/dtype errors), the utility wraps the failure in this RuntimeError with the model class and the input keys that were passed. The original exception is chained via 'from e'.

Source

Thrown at src/transformers/exporters/utils.py:767

    Returns:
        `dict[str, tuple[torch.nn.Module, dict]]`:
        `{"prefill": (model, prefill_inputs), "decode": (model, decode_inputs)}`.
    """
    # 1 prefill forward + 1 decode (or 2 decode steps merged, when `multi_token_decode`) forward to capture.
    # Set the capture window on the config itself, not as generate() kwargs — passing a
    # `generation_config` alongside generation kwargs is deprecated. Base it on the model's own config
    # when none is given (preserving its defaults), and deep-copy into a distinct `capture_config` so
    # the caller's `generation_config` is never mutated.
    num_new_tokens = 3 if multi_token_decode else 2
    capture_config = copy.deepcopy(generation_config if generation_config is not None else model.generation_config)
    capture_config.max_new_tokens = num_new_tokens
    capture_config.min_new_tokens = num_new_tokens
    try:
        with _capture_forward(model) as calls:
            model.generate(**copy.deepcopy(inputs), generation_config=capture_config)
    except Exception as e:
        raise RuntimeError(
            f"decompose_prefill_decode failed for {type(model).__name__}. "
            f"Inputs passed: {list(inputs.keys())}. "
            f"Make sure the inputs are compatible with model.generate()."
        ) from e

    if len(calls) < num_new_tokens:
        raise RuntimeError(
            f"decompose_prefill_decode expected at least {num_new_tokens} calls to "
            f"{type(model).__name__}.forward() during generate(max_new_tokens={num_new_tokens}), but "
            f"captured {len(calls)}. This likely means generate() bypasses the top-level forward() "
            "(e.g. delegates to an inner model), so prefill/decode decomposition is not supported "
            "for this architecture."
        )

    # Remove `logits_to_keep` from the captured calls — it's a generation-time hint for the model's
    # internal top-k pruning, not a forward input. The export graph should not depend on it.
    for call in calls:
        call.pop("logits_to_keep", None)

View on GitHub (pinned to a597f97485)

Solutions

  1. Reproduce outside export: call model.generate(**inputs) directly and read the chained __cause__ exception for the real error
  2. Fix the inputs dict so generate() accepts it (correct keys, tensors, device, dtype)
  3. Check model.generation_config (eos_token_id, pad_token_id) is valid for generation
  4. Only then re-run the export decomposition

Example fix

// before
export_packages = decompose_for_generation(model, {"raw_audio": wav})

// after
inputs = processor(wav, sampling_rate=16000, return_tensors="pt")
export_packages = decompose_for_generation(model, inputs)
Defensive patterns

Strategy: try-catch

Validate before calling

# Dry-run generate with the same inputs before export
def can_generate(model, inputs) -> bool:
    try:
        model.generate(**{k: v.clone() if hasattr(v, "clone") else v for k, v in inputs.items()}, max_new_tokens=1)
        return True
    except Exception:
        return False

Try / catch

try:
    decompose_for_generation(model, inputs)
except RuntimeError as e:
    cause = e.__cause__
    raise RuntimeError(f"export decomposition failed; root cause: {cause!r}") from cause

Prevention

When it happens

Trigger: Calling the export decomposition helper with an inputs dict incompatible with model.generate(): wrong key names, unprocessed raw features (e.g. float audio not yet a tensor), inputs on the wrong device, or a generation_config missing required fields.

Common situations: Exporting a decoder-only or multimodal model for deployment; passing preprocessor outputs instead of model-ready tensors; generate() failing due to eos/pad token misconfiguration that only surfaces at generation time.

Related errors

AI-assisted analysis of huggingface/transformers@a597f97485 (2026-08-14). Data as JSON: /api/errors/a40c9f140a9b3fd6. Report an issue: GitHub.