huggingface/transformers · error · ValueError

You provided `compile_config` as an instance of {}, but it m

Error message

You provided `compile_config` as an instance of {}, but it must be an instance of `CompileConfig`.

What it means

compile_config must be an instance of CompileConfig (the structured config for torch.compile of generation); passing a plain dict, string, or other object means generate() could not read its fields, so validate() rejects it.

Source

Thrown at src/transformers/generation/configuration_utils.py:698

                "`model.generation_config.pad_token_id=PAD_TOKEN_ID` to avoid errors in generation"
            )
        # 1.2. Cache attributes
        # "paged" re-routes to continuous batching and so it is a valid cache implementation. But we do not want to test
        # it with the `generate` as the other would be, so we we cannot add it to ALL_CACHE_IMPLEMENTATIONS
        valid_cache_implementations = ALL_CACHE_IMPLEMENTATIONS + ("paged",)
        if self.cache_implementation is not None and self.cache_implementation not in valid_cache_implementations:
            raise ValueError(
                f"Invalid `cache_implementation` ({self.cache_implementation}). Choose one of: "
                f"{valid_cache_implementations}"
            )
        if self.max_cache_len is not None and self.cache_implementation not in ALL_STATIC_CACHE_IMPLEMENTATIONS:
            logger.warning_once(
                f"`max_cache_len` is only used with static caches ({STATIC_CACHE_IMPLEMENTATIONS}); it will be "
                f"ignored with `cache_implementation={self.cache_implementation!r}`."
            )
        # 1.3. Performance attributes
        if self.compile_config is not None and not isinstance(self.compile_config, CompileConfig):
            raise ValueError(
                f"You provided `compile_config` as an instance of {type(self.compile_config)}, but it must be an "
                "instance of `CompileConfig`."
            )
        # 1.4. Watermarking attributes
        if self.watermarking_config is not None:
            self.watermarking_config.validate()

        # 2. Validation of attribute combinations
        # 2.1. detect sampling-only parameterization when not in sampling mode

        # Note that we check `is not True` in purpose. Boolean fields can also be `None` so we
        # have to be explicit. Value of `None` is same as having `False`, i.e. the default value

        if self.do_sample is not True:
            greedy_wrong_parameter_msg = (
                "`do_sample` is not set to `True`. However, `{flag_name}` is set to `{flag_value}` -- this flag is "
                "only used in sample-based generation modes. You should set `do_sample=True` or unset `{flag_name}`."
            )

View on GitHub (pinned to a597f97485)

Solutions

Wrap the options: GenerationConfig(compile_config=CompileConfig(mode='reduce-overhead'))
For a dict from JSON, convert: CompileConfig(**d)
Set compile_config=None if you do not want compiled generation

Example fix

# before
cfg = GenerationConfig(compile_config={"mode": "reduce-overhead", "fullgraph": True})
# after
from transformers import CompileConfig
cfg = GenerationConfig(compile_config=CompileConfig(mode="reduce-overhead", fullgraph=True))

Defensive patterns

Strategy: type-guard

Validate before calling

from transformers import CompileConfig

def coerce_compile_config(v):
    if v is None or isinstance(v, CompileConfig):
        return v
    if isinstance(v, dict):
        return CompileConfig(**v)
    raise TypeError(f"compile_config must be CompileConfig or dict, got {type(v)!r}")

Type guard

from transformers import CompileConfig

def is_compile_config(v) -> bool:
    return v is None or isinstance(v, CompileConfig)

Prevention

Construct CompileConfig explicitly instead of passing dicts
After loading generation_config.json, convert dict compile_config via CompileConfig(**d)

When it happens

Trigger: GenerationConfig(compile_config={'mode': 'reduce-overhead'}) or cache_implementation='compiled' with compile_config=True; or deserializing a generation_config.json whose compile_config arrives as a dict.

Common situations: Users writing compile options as dicts (natural from JSON), older configs with compile_config=True to toggle compilation, or merging configs from external tools.

Related errors

AI-assisted analysis of huggingface/transformers@a597f97485 (2026-08-14). Data as JSON: /api/errors/f9be0ee20c430cbc. Report an issue: GitHub.