huggingface/transformers · error · ValueError

A GenerationConfig must be provided or set in the model.

Error message

A GenerationConfig must be provided or set in the model.

What it means

Raised during continuous-batching manager init when no generation_config argument was passed and the model has generation_config = None. The manager needs sampling/EOS settings from a GenerationConfig and refuses to guess defaults beyond that.

Source

Thrown at src/transformers/generation/continuous_batching/continuous_api.py:1129

        """
        # Mandatory attributes
        if not hasattr(self, "config") or not hasattr(self, "device") or not hasattr(self, "dtype"):
            raise AttributeError("Model must have 'config', 'device', and 'dtype' attributes.")

        # If a persistent manager is found we return it
        cached_manager = getattr(self, "_cached_continuous_batching_manager", None)
        if isinstance(cached_manager, ContinuousBatchingManager):
            logger.info(
                "Cached continuous batching manager found: it will be re-used instead of creating a new one. If you"
                " want to create a new manager, you should call `destroy_cached_continuous_batching_manager` first."
            )
            cached_manager.switch_to_cb_friendly_attn(self)  # might have switched in .stop
            return cached_manager

        # Retrieve generation config
        gen_config = generation_config if generation_config is not None else self.generation_config
        if gen_config is None:
            raise ValueError("A GenerationConfig must be provided or set in the model.")
        # Warn about EOS
        if gen_config.eos_token_id is None:
            logger.warning("`eos_token_id` not set in GenerationConfig. Setting to -1 (disabled).")
            gen_config.eos_token_id = -1

        # Retrieve continuous batching config, or create it if none is provided
        if continuous_batching_config is None:
            if isinstance(getattr(gen_config, "continuous_batching_config", None), ContinuousBatchingConfig):
                logger.warning(
                    "Passing ContinuousBatchingConfig through GenerationConfig is deprecated. Please pass it separately"
                    " using the continuous_batching_config kwarg."
                )
                continuous_batching_config = gen_config.continuous_batching_config
            else:
                continuous_batching_config = ContinuousBatchingConfig()

        # Create and return the manager
        return ContinuousBatchingManager(

View on GitHub (pinned to a597f97485)

Solutions

1. Pass an explicit GenerationConfig: model.continuous_batching(generation_config=GenerationConfig(...))
2. Set model.generation_config = GenerationConfig.from_pretrained(checkpoint) before the call
3. For minimal use, GenerationConfig(eos_token_id=tokenizer.eos_token_id) satisfies the requirement

Example fix

# before
manager = model.continuous_batching()  # model.generation_config is None

# after
from transformers import GenerationConfig
manager = model.continuous_batching(generation_config=GenerationConfig(eos_token_id=tokenizer.eos_token_id))

gen_cfg = generation_config or getattr(model, 'generation_config', None)
if gen_cfg is None:
    from transformers import GenerationConfig
    gen_cfg = GenerationConfig(eos_token_id=tokenizer.eos_token_id)
    model.generation_config = gen_cfg

from transformers import GenerationConfig
def has_generation_config(model) -> bool:
    return getattr(model, 'generation_config', None) is not None or isinstance(getattr(model, 'generation_config', None), GenerationConfig)

When it happens

Trigger: model.generation_config is None (some freshly assembled from_config/torchscript models, or users who explicitly set it to None) and the caller omits generation_config in the continuous_batching call.

Common situations: Loading sharded/converted checkpoints that skip generation_config creation; manually constructing a model then deleting generation_config; library code that passes generation_config conditionally.

Related errors

• Unknown logit_processor_kwargs: {unknown_keys}. {self.suppor
• `early_stopping` must be a boolean or 'never', but is {}.
• `max_new_tokens` must be greater than 0, but is {}.
• Invalid `cache_implementation` ({}). Choose one of: {}
• Greedy methods (do_sample != True) without beam search do no

AI-assisted analysis of huggingface/transformers@a597f97485 (2026-08-14). Data as JSON: /api/errors/853d239f06956b79. Report an issue: GitHub.