huggingface/transformers · error · AttributeError

Model must have 'config', 'device', and 'dtype' attributes.

Error message

Model must have 'config', 'device', and 'dtype' attributes.

What it means

Raised at the top of continuous-batching manager initialization: the object it is attached to must expose config, device, and dtype attributes. It is an AttributeError guard ensuring the API is called on a model-like object, not a bare nn.Module, a function output, or a misplaced call.

Source

Thrown at src/transformers/generation/continuous_batching/continuous_api.py:1114

    def init_continuous_batching(
        self,
        generation_config: GenerationConfig | None = None,
        continuous_batching_config: ContinuousBatchingConfig | None = None,
        workload_hints: WorkloadHints | None = None,
    ) -> ContinuousBatchingManager:
        """Initialize a manager for continuous batching inference.

        Args:
            generation_config: An optional generation configuration, which may contain a CompileConfig object
            continuous_batching_config: An optional continuous batching configuration
            workload_hints: Optional WorkloadHints to help the continuous batching manager make better decisions for
                default values
        Returns:
            `ContinuousBatchingManager`: The manager instance to add requests and retrieve results.
        """
        # Mandatory attributes
        if not hasattr(self, "config") or not hasattr(self, "device") or not hasattr(self, "dtype"):
            raise AttributeError("Model must have 'config', 'device', and 'dtype' attributes.")

        # If a persistent manager is found we return it
        cached_manager = getattr(self, "_cached_continuous_batching_manager", None)
        if isinstance(cached_manager, ContinuousBatchingManager):
            logger.info(
                "Cached continuous batching manager found: it will be re-used instead of creating a new one. If you"
                " want to create a new manager, you should call `destroy_cached_continuous_batching_manager` first."
            )
            cached_manager.switch_to_cb_friendly_attn(self)  # might have switched in .stop
            return cached_manager

        # Retrieve generation config
        gen_config = generation_config if generation_config is not None else self.generation_config
        if gen_config is None:
            raise ValueError("A GenerationConfig must be provided or set in the model.")
        # Warn about EOS
        if gen_config.eos_token_id is None:
            logger.warning("`eos_token_id` not set in GenerationConfig. Setting to -1 (disabled).")

View on GitHub (pinned to a597f97485)

Solutions

1. Call the API on the full PreTrainedModel instance, not a submodule or wrapper
2. If using a wrapper, expose config/device/dtype properties delegating to the inner model
3. Check the call site order of arguments — a swapped model/tokenizer argument triggers this

Example fix

# before
manager = my_custom_wrapper.continuous_batching()  # wrapper lacks attrs

# after
class MyWrapper:
    @property
    def config(self): return self.model.config
    @property
    def device(self): return self.model.device
    @property
    def dtype(self): return self.model.dtype
manager = my_custom_wrapper.continuous_batching()

for attr in ('config', 'device', 'dtype'):
    assert hasattr(model, attr), f'model lacks .{attr} — pass the full PreTrainedModel'

from transformers import PreTrainedModel
def is_cb_capable(obj) -> bool:
    return isinstance(obj, PreTrainedModel) or all(hasattr(obj, a) for a in ('config', 'device', 'dtype'))

When it happens

Trigger: Calling model.continuous_batching(...) on a wrapper/module lacking PreTrainedModel attributes; calling the free function continuous_batching(obj) with obj = some submodule (e.g. model.decoder) or a Pipeline/optimizer by accident.

Common situations: Passing a torch.compile'd or wrapped model; using a custom inference wrapper class that forwards generate() but not config/device/dtype; typo passing the tokenizer as the model.

Related errors

• Audio transcription requires sequential generation (not CB)
• CB worker is dead and cannot accept request {request_id}: {s
• CB worker died during request {request_id}: {result.error}
• Cannot assign to field {name}, you should create a new insta
• Framework '{return_tensors}' not recognized!

AI-assisted analysis of huggingface/transformers@a597f97485 (2026-08-14). Data as JSON: /api/errors/2c875ef377f273ea. Report an issue: GitHub.