huggingface/transformers · error · ValueError

`assistant_tokenizer` is not required when the main and assi

Error message

`assistant_tokenizer` is not required when the main and assistant models use the same tokenizer. Please omit `assistant_tokenizer` from `generate()` {doc_reference}.

What it means

In universal assisted decoding, transformers decides whether main and assistant share a tokenizer by comparing `vocab_size` of the two text configs. When the vocab sizes are EQUAL, passing `assistant_tokenizer` is contradictory and rejected — the assistant's own tokenizer is already correct and a second one invites mismatches.

Source

Thrown at src/transformers/generation/utils.py:1605

        ) is not None and generation_config.speculation_type != "dflash":
            if self.config.is_encoder_decoder and not assistant_model.config.is_encoder_decoder:
                attributes_to_check = ["encoder_attention_heads", "encoder_ffn_dim", "encoder_layers"]
                attributes_to_check = [attr for attr in dir(assistant_model.config) if attr in attributes_to_check]
                are_equal = all(
                    getattr(self.config, attr) == getattr(assistant_model.config, attr) for attr in attributes_to_check
                )
                if not are_equal:
                    raise ValueError(
                        "The main model and the assistant don't have compatible encoder-dependent input shapes. "
                        "Ensure you load the assistant with the correct encoder-decoder class, e.g. `AutoModelForSpeechSeq2Seq` for Whisper."
                    )

            doc_reference = (
                "(see https://huggingface.co/docs/transformers/en/generation_strategies#universal-assisted-decoding)"
            )
            if self.config.get_text_config().vocab_size == assistant_model.config.get_text_config().vocab_size:
                if "assistant_tokenizer" in generation_mode_kwargs:
                    raise ValueError(
                        f"`assistant_tokenizer` is not required when the main and assistant models use the same tokenizer. Please omit `assistant_tokenizer` from `generate()` {doc_reference}."
                    )
            else:
                if "tokenizer" not in generation_mode_kwargs or "assistant_tokenizer" not in generation_mode_kwargs:
                    raise ValueError(
                        f"The main and assistant models have different tokenizers. Please provide `tokenizer` and `assistant_tokenizer` to `generate()` {doc_reference}."
                    )

    def _validate_model_kwargs(self: "GenerativePreTrainedModel", model_kwargs: dict[str, Any]):
        """Validates model kwargs for generation. Generate argument typos will also be caught here."""
        # Excludes arguments that are handled before calling any model function
        if self.config.is_encoder_decoder:
            for key in ["decoder_input_ids"]:
                model_kwargs.pop(key, None)

        unused_model_args = []
        model_args = set(inspect.signature(self.prepare_inputs_for_generation).parameters)
        # `kwargs`/`model_kwargs` is often used to handle optional forward pass inputs like `attention_mask`. If

View on GitHub (pinned to a597f97485)

Solutions

Omit `assistant_tokenizer` from the generate call — pass only `tokenizer`.
Confirm the pair truly shares a tokenizer: compare `model.config.get_text_config().vocab_size` and `assistant.config.get_text_config().vocab_size`.
If you did NOT intend a same-tokenizer pair, check that you loaded the intended assistant checkpoint.

Example fix

# before
out = model.generate(
    **inputs, assistant_model=assistant,
    tokenizer=tokenizer, assistant_tokenizer=assistant_tok,  # vocab sizes equal -> ValueError
)

# after
out = model.generate(**inputs, assistant_model=assistant, tokenizer=tokenizer)

Defensive patterns

Strategy: validation

Validate before calling

same_vocab = model.config.get_text_config().vocab_size == assistant.config.get_text_config().vocab_size
if same_vocab:
    kwargs.pop("assistant_tokenizer", None)  # not required and will raise

Prevention

Pass assistant_tokenizer ONLY when the two vocab sizes actually differ.
Compare vocab_size before building generate kwargs in cross-model pipelines.
Follow the universal assisted decoding doc for the exact token combination rule.

When it happens

Trigger: `model.generate(..., assistant_model=assistant, assistant_tokenizer=other_tok)` where `model.config.get_text_config().vocab_size == assistant.config.get_text_config().vocab_size` (same tokenizer family).

Common situations: Copy-pasting the universal-assisted-decoding example (written for different-tokenizer pairs) onto a same-tokenizer pair like a model plus its own distilled version; defensively passing both tokenizers 'just in case'.

Related errors

AI-assisted analysis of huggingface/transformers@a597f97485 (2026-08-14). Data as JSON: /api/errors/286a6afc99cdd290. Report an issue: GitHub.