huggingface/transformers · error · ValueError

The main and assistant models have different tokenizers. Ple

Error message

The main and assistant models have different tokenizers. Please provide `tokenizer` and `assistant_tokenizer` to `generate()` {doc_reference}.

What it means

The mirror case of the same-tokenizer check: when the main and assistant text configs have DIFFERENT `vocab_size`, they necessarily use different tokenizers, and universal assisted decoding needs both to map tokens between the two spaces. `generate` raises when `tokenizer` and/or `assistant_tokenizer` is missing from the call.

Source

Thrown at src/transformers/generation/utils.py:1610

                    getattr(self.config, attr) == getattr(assistant_model.config, attr) for attr in attributes_to_check
                )
                if not are_equal:
                    raise ValueError(
                        "The main model and the assistant don't have compatible encoder-dependent input shapes. "
                        "Ensure you load the assistant with the correct encoder-decoder class, e.g. `AutoModelForSpeechSeq2Seq` for Whisper."
                    )

            doc_reference = (
                "(see https://huggingface.co/docs/transformers/en/generation_strategies#universal-assisted-decoding)"
            )
            if self.config.get_text_config().vocab_size == assistant_model.config.get_text_config().vocab_size:
                if "assistant_tokenizer" in generation_mode_kwargs:
                    raise ValueError(
                        f"`assistant_tokenizer` is not required when the main and assistant models use the same tokenizer. Please omit `assistant_tokenizer` from `generate()` {doc_reference}."
                    )
            else:
                if "tokenizer" not in generation_mode_kwargs or "assistant_tokenizer" not in generation_mode_kwargs:
                    raise ValueError(
                        f"The main and assistant models have different tokenizers. Please provide `tokenizer` and `assistant_tokenizer` to `generate()` {doc_reference}."
                    )

    def _validate_model_kwargs(self: "GenerativePreTrainedModel", model_kwargs: dict[str, Any]):
        """Validates model kwargs for generation. Generate argument typos will also be caught here."""
        # Excludes arguments that are handled before calling any model function
        if self.config.is_encoder_decoder:
            for key in ["decoder_input_ids"]:
                model_kwargs.pop(key, None)

        unused_model_args = []
        model_args = set(inspect.signature(self.prepare_inputs_for_generation).parameters)
        # `kwargs`/`model_kwargs` is often used to handle optional forward pass inputs like `attention_mask`. If
        # `prepare_inputs_for_generation` doesn't accept them, then a stricter check can be made ;)
        if "kwargs" in model_args or "model_kwargs" in model_args:
            model_args |= set(inspect.signature(self.forward).parameters)

        # Encoder-Decoder models may also need Encoder arguments from `model_kwargs`

View on GitHub (pinned to a597f97485)

Solutions

Pass both: `model.generate(**tokenizer(prompt, return_tensors="pt"), assistant_model=assistant, tokenizer=tokenizer, assistant_tokenizer=assistant_tokenizer)`.
Ensure `assistant_tokenizer` is the tokenizer the assistant checkpoint was trained with, not another copy of the main one.
If you expected same-tokenizer behavior, verify which assistant checkpoint you loaded (its vocab_size differs from the main model).

Example fix

# before
out = model.generate(**inputs, assistant_model=assistant)  # different vocab_size, no tokenizers -> ValueError

# after
out = model.generate(
    **inputs,
    assistant_model=assistant,
    tokenizer=tokenizer,
    assistant_tokenizer=assistant_tokenizer,
)

Defensive patterns

Strategy: validation

Validate before calling

if model.config.get_text_config().vocab_size != assistant.config.get_text_config().vocab_size:
    for key in ("tokenizer", "assistant_tokenizer"):
        if key not in kwargs:
            raise ValueError(f"{key} required: main and assistant use different tokenizers")

Prevention

When pairing cross-family models, always pass both tokenizer and assistant_tokenizer.
Wrap assisted generation in a helper that checks vocab sizes and injects the right tokenizers.
Use each checkpoint's own tokenizer (from_pretrained per model), never one tokenizer for both.

When it happens

Trigger: `model.generate(**inputs, assistant_model=assistant)` where the two `vocab_size`s differ and one or both of `tokenizer`/`assistant_tokenizer` kwargs are absent — e.g. passing only `input_ids` prepared with the main tokenizer.

Common situations: Pairing models from different families (e.g. a main model and an assistant with a different tokenizer/vocab); following the assisted-decoding quickstart (same-tokenizer) and swapping in a cross-family assistant without adding both tokenizers; tokenizing inputs manually and forgetting kwargs get forwarded.

Related errors

AI-assisted analysis of huggingface/transformers@a597f97485 (2026-08-14). Data as JSON: /api/errors/6cb42b15d2496ffd. Report an issue: GitHub.